thesis learning technology

REALIZING THE PROMISE:

Leading up to the 75th anniversary of the UN General Assembly, this “Realizing the promise: How can education technology improve learning for all?” publication kicks off the Center for Universal Education’s first playbook in a series to help improve education around the world.

It is intended as an evidence-based tool for ministries of education, particularly in low- and middle-income countries, to adopt and more successfully invest in education technology.

While there is no single education initiative that will achieve the same results everywhere—as school systems differ in learners and educators, as well as in the availability and quality of materials and technologies—an important first step is understanding how technology is used given specific local contexts and needs.

The surveys in this playbook are designed to be adapted to collect this information from educators, learners, and school leaders and guide decisionmakers in expanding the use of technology.  

Introduction

While technology has disrupted most sectors of the economy and changed how we communicate, access information, work, and even play, its impact on schools, teaching, and learning has been much more limited. We believe that this limited impact is primarily due to technology being been used to replace analog tools, without much consideration given to playing to technology’s comparative advantages. These comparative advantages, relative to traditional “chalk-and-talk” classroom instruction, include helping to scale up standardized instruction, facilitate differentiated instruction, expand opportunities for practice, and increase student engagement. When schools use technology to enhance the work of educators and to improve the quality and quantity of educational content, learners will thrive.

Further, COVID-19 has laid bare that, in today’s environment where pandemics and the effects of climate change are likely to occur, schools cannot always provide in-person education—making the case for investing in education technology.

Here we argue for a simple yet surprisingly rare approach to education technology that seeks to:

• Understand the needs, infrastructure, and capacity of a school system—the diagnosis;
• Survey the best available evidence on interventions that match those conditions—the evidence; and
• Closely monitor the results of innovations before they are scaled up—the prognosis.

RELATED CONTENT

Podcast: How education technology can improve learning for all students

To make ed tech work, set clear goals, review the evidence, and pilot before you scale

The framework.

Our approach builds on a simple yet intuitive theoretical framework created two decades ago by two of the most prominent education researchers in the United States, David K. Cohen and Deborah Loewenberg Ball. They argue that what matters most to improve learning is the interactions among educators and learners around educational materials. We believe that the failed school-improvement efforts in the U.S. that motivated Cohen and Ball’s framework resemble the ed-tech reforms in much of the developing world to date in the lack of clarity improving the interactions between educators, learners, and the educational material. We build on their framework by adding parents as key agents that mediate the relationships between learners and educators and the material (Figure 1).

Figure 1: The instructional core

Adapted from Cohen and Ball (1999)

As the figure above suggests, ed-tech interventions can affect the instructional core in a myriad of ways. Yet, just because technology can do something, it does not mean it should. School systems in developing countries differ along many dimensions and each system is likely to have different needs for ed-tech interventions, as well as different infrastructure and capacity to enact such interventions.

The diagnosis:

How can school systems assess their needs and preparedness.

A useful first step for any school system to determine whether it should invest in education technology is to diagnose its:

• Specific needs to improve student learning (e.g., raising the average level of achievement, remediating gaps among low performers, and challenging high performers to develop higher-order skills);
• Infrastructure to adopt technology-enabled solutions (e.g., electricity connection, availability of space and outlets, stock of computers, and Internet connectivity at school and at learners’ homes); and
• Capacity to integrate technology in the instructional process (e.g., learners’ and educators’ level of familiarity and comfort with hardware and software, their beliefs about the level of usefulness of technology for learning purposes, and their current uses of such technology).

Before engaging in any new data collection exercise, school systems should take full advantage of existing administrative data that could shed light on these three main questions. This could be in the form of internal evaluations but also international learner assessments, such as the Program for International Student Assessment (PISA), the Trends in International Mathematics and Science Study (TIMSS), and/or the Progress in International Literacy Study (PIRLS), and the Teaching and Learning International Study (TALIS). But if school systems lack information on their preparedness for ed-tech reforms or if they seek to complement existing data with a richer set of indicators, we developed a set of surveys for learners, educators, and school leaders. Download the full report to see how we map out the main aspects covered by these surveys, in hopes of highlighting how they could be used to inform decisions around the adoption of ed-tech interventions.

The evidence:

How can school systems identify promising ed-tech interventions.

There is no single “ed-tech” initiative that will achieve the same results everywhere, simply because school systems differ in learners and educators, as well as in the availability and quality of materials and technologies. Instead, to realize the potential of education technology to accelerate student learning, decisionmakers should focus on four potential uses of technology that play to its comparative advantages and complement the work of educators to accelerate student learning (Figure 2). These comparative advantages include:

• Scaling up quality instruction, such as through prerecorded quality lessons.
• Facilitating differentiated instruction, through, for example, computer-adaptive learning and live one-on-one tutoring.
• Expanding opportunities to practice.
• Increasing learner engagement through videos and games.

Figure 2: Comparative advantages of technology

Here we review the evidence on ed-tech interventions from 37 studies in 20 countries*, organizing them by comparative advantage. It’s important to note that ours is not the only way to classify these interventions (e.g., video tutorials could be considered as a strategy to scale up instruction or increase learner engagement), but we believe it may be useful to highlight the needs that they could address and why technology is well positioned to do so.

When discussing specific studies, we report the magnitude of the effects of interventions using standard deviations (SDs). SDs are a widely used metric in research to express the effect of a program or policy with respect to a business-as-usual condition (e.g., test scores). There are several ways to make sense of them. One is to categorize the magnitude of the effects based on the results of impact evaluations. In developing countries, effects below 0.1 SDs are considered to be small, effects between 0.1 and 0.2 SDs are medium, and those above 0.2 SDs are large (for reviews that estimate the average effect of groups of interventions, called “meta analyses,” see e.g., Conn, 2017; Kremer, Brannen, & Glennerster, 2013; McEwan, 2014; Snilstveit et al., 2015; Evans & Yuan, 2020.)

*In surveying the evidence, we began by compiling studies from prior general and ed-tech specific evidence reviews that some of us have written and from ed-tech reviews conducted by others. Then, we tracked the studies cited by the ones we had previously read and reviewed those, as well. In identifying studies for inclusion, we focused on experimental and quasi-experimental evaluations of education technology interventions from pre-school to secondary school in low- and middle-income countries that were released between 2000 and 2020. We only included interventions that sought to improve student learning directly (i.e., students’ interaction with the material), as opposed to interventions that have impacted achievement indirectly, by reducing teacher absence or increasing parental engagement. This process yielded 37 studies in 20 countries (see the full list of studies in Appendix B).

Scaling up standardized instruction

One of the ways in which technology may improve the quality of education is through its capacity to deliver standardized quality content at scale. This feature of technology may be particularly useful in three types of settings: (a) those in “hard-to-staff” schools (i.e., schools that struggle to recruit educators with the requisite training and experience—typically, in rural and/or remote areas) (see, e.g., Urquiola & Vegas, 2005); (b) those in which many educators are frequently absent from school (e.g., Chaudhury, Hammer, Kremer, Muralidharan, & Rogers, 2006; Muralidharan, Das, Holla, & Mohpal, 2017); and/or (c) those in which educators have low levels of pedagogical and subject matter expertise (e.g., Bietenbeck, Piopiunik, & Wiederhold, 2018; Bold et al., 2017; Metzler & Woessmann, 2012; Santibañez, 2006) and do not have opportunities to observe and receive feedback (e.g., Bruns, Costa, & Cunha, 2018; Cilliers, Fleisch, Prinsloo, & Taylor, 2018). Technology could address this problem by: (a) disseminating lessons delivered by qualified educators to a large number of learners (e.g., through prerecorded or live lessons); (b) enabling distance education (e.g., for learners in remote areas and/or during periods of school closures); and (c) distributing hardware preloaded with educational materials.

Prerecorded lessons

Technology seems to be well placed to amplify the impact of effective educators by disseminating their lessons. Evidence on the impact of prerecorded lessons is encouraging, but not conclusive. Some initiatives that have used short instructional videos to complement regular instruction, in conjunction with other learning materials, have raised student learning on independent assessments. For example, Beg et al. (2020) evaluated an initiative in Punjab, Pakistan in which grade 8 classrooms received an intervention that included short videos to substitute live instruction, quizzes for learners to practice the material from every lesson, tablets for educators to learn the material and follow the lesson, and LED screens to project the videos onto a classroom screen. After six months, the intervention improved the performance of learners on independent tests of math and science by 0.19 and 0.24 SDs, respectively but had no discernible effect on the math and science section of Punjab’s high-stakes exams.

One study suggests that approaches that are far less technologically sophisticated can also improve learning outcomes—especially, if the business-as-usual instruction is of low quality. For example, Naslund-Hadley, Parker, and Hernandez-Agramonte (2014) evaluated a preschool math program in Cordillera, Paraguay that used audio segments and written materials four days per week for an hour per day during the school day. After five months, the intervention improved math scores by 0.16 SDs, narrowing gaps between low- and high-achieving learners, and between those with and without educators with formal training in early childhood education.

Yet, the integration of prerecorded material into regular instruction has not always been successful. For example, de Barros (2020) evaluated an intervention that combined instructional videos for math and science with infrastructure upgrades (e.g., two “smart” classrooms, two TVs, and two tablets), printed workbooks for students, and in-service training for educators of learners in grades 9 and 10 in Haryana, India (all materials were mapped onto the official curriculum). After 11 months, the intervention negatively impacted math achievement (by 0.08 SDs) and had no effect on science (with respect to business as usual classes). It reduced the share of lesson time that educators devoted to instruction and negatively impacted an index of instructional quality. Likewise, Seo (2017) evaluated several combinations of infrastructure (solar lights and TVs) and prerecorded videos (in English and/or bilingual) for grade 11 students in northern Tanzania and found that none of the variants improved student learning, even when the videos were used. The study reports effects from the infrastructure component across variants, but as others have noted (Muralidharan, Romero, & Wüthrich, 2019), this approach to estimating impact is problematic.

A very similar intervention delivered after school hours, however, had sizeable effects on learners’ basic skills. Chiplunkar, Dhar, and Nagesh (2020) evaluated an initiative in Chennai (the capital city of the state of Tamil Nadu, India) delivered by the same organization as above that combined short videos that explained key concepts in math and science with worksheets, facilitator-led instruction, small groups for peer-to-peer learning, and occasional career counseling and guidance for grade 9 students. These lessons took place after school for one hour, five times a week. After 10 months, it had large effects on learners’ achievement as measured by tests of basic skills in math and reading, but no effect on a standardized high-stakes test in grade 10 or socio-emotional skills (e.g., teamwork, decisionmaking, and communication).

Drawing general lessons from this body of research is challenging for at least two reasons. First, all of the studies above have evaluated the impact of prerecorded lessons combined with several other components (e.g., hardware, print materials, or other activities). Therefore, it is possible that the effects found are due to these additional components, rather than to the recordings themselves, or to the interaction between the two (see Muralidharan, 2017 for a discussion of the challenges of interpreting “bundled” interventions). Second, while these studies evaluate some type of prerecorded lessons, none examines the content of such lessons. Thus, it seems entirely plausible that the direction and magnitude of the effects depends largely on the quality of the recordings (e.g., the expertise of the educator recording it, the amount of preparation that went into planning the recording, and its alignment with best teaching practices).

These studies also raise three important questions worth exploring in future research. One of them is why none of the interventions discussed above had effects on high-stakes exams, even if their materials are typically mapped onto the official curriculum. It is possible that the official curricula are simply too challenging for learners in these settings, who are several grade levels behind expectations and who often need to reinforce basic skills (see Pritchett & Beatty, 2015). Another question is whether these interventions have long-term effects on teaching practices. It seems plausible that, if these interventions are deployed in contexts with low teaching quality, educators may learn something from watching the videos or listening to the recordings with learners. Yet another question is whether these interventions make it easier for schools to deliver instruction to learners whose native language is other than the official medium of instruction.

Distance education

Technology can also allow learners living in remote areas to access education. The evidence on these initiatives is encouraging. For example, Johnston and Ksoll (2017) evaluated a program that broadcasted live instruction via satellite to rural primary school students in the Volta and Greater Accra regions of Ghana. For this purpose, the program also equipped classrooms with the technology needed to connect to a studio in Accra, including solar panels, a satellite modem, a projector, a webcam, microphones, and a computer with interactive software. After two years, the intervention improved the numeracy scores of students in grades 2 through 4, and some foundational literacy tasks, but it had no effect on attendance or classroom time devoted to instruction, as captured by school visits. The authors interpreted these results as suggesting that the gains in achievement may be due to improving the quality of instruction that children received (as opposed to increased instructional time). Naik, Chitre, Bhalla, and Rajan (2019) evaluated a similar program in the Indian state of Karnataka and also found positive effects on learning outcomes, but it is not clear whether those effects are due to the program or due to differences in the groups of students they compared to estimate the impact of the initiative.

In one context (Mexico), this type of distance education had positive long-term effects. Navarro-Sola (2019) took advantage of the staggered rollout of the telesecundarias (i.e., middle schools with lessons broadcasted through satellite TV) in 1968 to estimate its impact. The policy had short-term effects on students’ enrollment in school: For every telesecundaria per 50 children, 10 students enrolled in middle school and two pursued further education. It also had a long-term influence on the educational and employment trajectory of its graduates. Each additional year of education induced by the policy increased average income by nearly 18 percent. This effect was attributable to more graduates entering the labor force and shifting from agriculture and the informal sector. Similarly, Fabregas (2019) leveraged a later expansion of this policy in 1993 and found that each additional telesecundaria per 1,000 adolescents led to an average increase of 0.2 years of education, and a decline in fertility for women, but no conclusive evidence of long-term effects on labor market outcomes.

It is crucial to interpret these results keeping in mind the settings where the interventions were implemented. As we mention above, part of the reason why they have proven effective is that the “counterfactual” conditions for learning (i.e., what would have happened to learners in the absence of such programs) was either to not have access to schooling or to be exposed to low-quality instruction. School systems interested in taking up similar interventions should assess the extent to which their learners (or parts of their learner population) find themselves in similar conditions to the subjects of the studies above. This illustrates the importance of assessing the needs of a system before reviewing the evidence.

Preloaded hardware

Technology also seems well positioned to disseminate educational materials. Specifically, hardware (e.g., desktop computers, laptops, or tablets) could also help deliver educational software (e.g., word processing, reference texts, and/or games). In theory, these materials could not only undergo a quality assurance review (e.g., by curriculum specialists and educators), but also draw on the interactions with learners for adjustments (e.g., identifying areas needing reinforcement) and enable interactions between learners and educators.

In practice, however, most initiatives that have provided learners with free computers, laptops, and netbooks do not leverage any of the opportunities mentioned above. Instead, they install a standard set of educational materials and hope that learners find them helpful enough to take them up on their own. Students rarely do so, and instead use the laptops for recreational purposes—often, to the detriment of their learning (see, e.g., Malamud & Pop-Eleches, 2011). In fact, free netbook initiatives have not only consistently failed to improve academic achievement in math or language (e.g., Cristia et al., 2017), but they have had no impact on learners’ general computer skills (e.g., Beuermann et al., 2015). Some of these initiatives have had small impacts on cognitive skills, but the mechanisms through which those effects occurred remains unclear.

To our knowledge, the only successful deployment of a free laptop initiative was one in which a team of researchers equipped the computers with remedial software. Mo et al. (2013) evaluated a version of the One Laptop per Child (OLPC) program for grade 3 students in migrant schools in Beijing, China in which the laptops were loaded with a remedial software mapped onto the national curriculum for math (similar to the software products that we discuss under “practice exercises” below). After nine months, the program improved math achievement by 0.17 SDs and computer skills by 0.33 SDs. If a school system decides to invest in free laptops, this study suggests that the quality of the software on the laptops is crucial.

To date, however, the evidence suggests that children do not learn more from interacting with laptops than they do from textbooks. For example, Bando, Gallego, Gertler, and Romero (2016) compared the effect of free laptop and textbook provision in 271 elementary schools in disadvantaged areas of Honduras. After seven months, students in grades 3 and 6 who had received the laptops performed on par with those who had received the textbooks in math and language. Further, even if textbooks essentially become obsolete at the end of each school year, whereas laptops can be reloaded with new materials for each year, the costs of laptop provision (not just the hardware, but also the technical assistance, Internet, and training associated with it) are not yet low enough to make them a more cost-effective way of delivering content to learners.

Evidence on the provision of tablets equipped with software is encouraging but limited. For example, de Hoop et al. (2020) evaluated a composite intervention for first grade students in Zambia’s Eastern Province that combined infrastructure (electricity via solar power), hardware (projectors and tablets), and educational materials (lesson plans for educators and interactive lessons for learners, both loaded onto the tablets and mapped onto the official Zambian curriculum). After 14 months, the intervention had improved student early-grade reading by 0.4 SDs, oral vocabulary scores by 0.25 SDs, and early-grade math by 0.22 SDs. It also improved students’ achievement by 0.16 on a locally developed assessment. The multifaceted nature of the program, however, makes it challenging to identify the components that are driving the positive effects. Pitchford (2015) evaluated an intervention that provided tablets equipped with educational “apps,” to be used for 30 minutes per day for two months to develop early math skills among students in grades 1 through 3 in Lilongwe, Malawi. The evaluation found positive impacts in math achievement, but the main study limitation is that it was conducted in a single school.

Facilitating differentiated instruction

Another way in which technology may improve educational outcomes is by facilitating the delivery of differentiated or individualized instruction. Most developing countries massively expanded access to schooling in recent decades by building new schools and making education more affordable, both by defraying direct costs, as well as compensating for opportunity costs (Duflo, 2001; World Bank, 2018). These initiatives have not only rapidly increased the number of learners enrolled in school, but have also increased the variability in learner’ preparation for schooling. Consequently, a large number of learners perform well below grade-based curricular expectations (see, e.g., Duflo, Dupas, & Kremer, 2011; Pritchett & Beatty, 2015). These learners are unlikely to get much from “one-size-fits-all” instruction, in which a single educator delivers instruction deemed appropriate for the middle (or top) of the achievement distribution (Banerjee & Duflo, 2011). Technology could potentially help these learners by providing them with: (a) instruction and opportunities for practice that adjust to the level and pace of preparation of each individual (known as “computer-adaptive learning” (CAL)); or (b) live, one-on-one tutoring.

Computer-adaptive learning

One of the main comparative advantages of technology is its ability to diagnose students’ initial learning levels and assign students to instruction and exercises of appropriate difficulty. No individual educator—no matter how talented—can be expected to provide individualized instruction to all learners in his/her class simultaneously . In this respect, technology is uniquely positioned to complement traditional teaching. This use of technology could help learners master basic skills and help them get more out of schooling.

Although many software products evaluated in recent years have been categorized as CAL, many rely on a relatively coarse level of differentiation at an initial stage (e.g., a diagnostic test) without further differentiation. We discuss these initiatives under the category of “increasing opportunities for practice” below. CAL initiatives complement an initial diagnostic with dynamic adaptation (i.e., at each response or set of responses from learners) to adjust both the initial level of difficulty and rate at which it increases or decreases, depending on whether learners’ responses are correct or incorrect.

Existing evidence on this specific type of programs is highly promising. Most famously, Banerjee et al. (2007) evaluated CAL software in Vadodara, in the Indian state of Gujarat, in which grade 4 students were offered two hours of shared computer time per week before and after school, during which they played games that involved solving math problems. The level of difficulty of such problems adjusted based on students’ answers. This program improved math achievement by 0.35 and 0.47 SDs after one and two years of implementation, respectively. Consistent with the promise of personalized learning, the software improved achievement for all students. In fact, one year after the end of the program, students assigned to the program still performed 0.1 SDs better than those assigned to a business as usual condition. More recently, Muralidharan, et al. (2019) evaluated a “blended learning” initiative in which students in grades 4 through 9 in Delhi, India received 45 minutes of interaction with CAL software for math and language, and 45 minutes of small group instruction before or after going to school. After only 4.5 months, the program improved achievement by 0.37 SDs in math and 0.23 SDs in Hindi. While all learners benefited from the program in absolute terms, the lowest performing learners benefited the most in relative terms, since they were learning very little in school.

We see two important limitations from this body of research. First, to our knowledge, none of these initiatives has been evaluated when implemented during the school day. Therefore, it is not possible to distinguish the effect of the adaptive software from that of additional instructional time. Second, given that most of these programs were facilitated by local instructors, attempts to distinguish the effect of the software from that of the instructors has been mostly based on noncausal evidence. A frontier challenge in this body of research is to understand whether CAL software can increase the effectiveness of school-based instruction by substituting part of the regularly scheduled time for math and language instruction.

Live one-on-one tutoring

Recent improvements in the speed and quality of videoconferencing, as well as in the connectivity of remote areas, have enabled yet another way in which technology can help personalization: live (i.e., real-time) one-on-one tutoring. While the evidence on in-person tutoring is scarce in developing countries, existing studies suggest that this approach works best when it is used to personalize instruction (see, e.g., Banerjee et al., 2007; Banerji, Berry, & Shotland, 2015; Cabezas, Cuesta, & Gallego, 2011).

There are almost no studies on the impact of online tutoring—possibly, due to the lack of hardware and Internet connectivity in low- and middle-income countries. One exception is Chemin and Oledan (2020)’s recent evaluation of an online tutoring program for grade 6 students in Kianyaga, Kenya to learn English from volunteers from a Canadian university via Skype ( videoconferencing software) for one hour per week after school. After 10 months, program beneficiaries performed 0.22 SDs better in a test of oral comprehension, improved their comfort using technology for learning, and became more willing to engage in cross-cultural communication. Importantly, while the tutoring sessions used the official English textbooks and sought in part to help learners with their homework, tutors were trained on several strategies to teach to each learner’s individual level of preparation, focusing on basic skills if necessary. To our knowledge, similar initiatives within a country have not yet been rigorously evaluated.

Expanding opportunities for practice

A third way in which technology may improve the quality of education is by providing learners with additional opportunities for practice. In many developing countries, lesson time is primarily devoted to lectures, in which the educator explains the topic and the learners passively copy explanations from the blackboard. This setup leaves little time for in-class practice. Consequently, learners who did not understand the explanation of the material during lecture struggle when they have to solve homework assignments on their own. Technology could potentially address this problem by allowing learners to review topics at their own pace.

Practice exercises

Technology can help learners get more out of traditional instruction by providing them with opportunities to implement what they learn in class. This approach could, in theory, allow some learners to anchor their understanding of the material through trial and error (i.e., by realizing what they may not have understood correctly during lecture and by getting better acquainted with special cases not covered in-depth in class).

Existing evidence on practice exercises reflects both the promise and the limitations of this use of technology in developing countries. For example, Lai et al. (2013) evaluated a program in Shaanxi, China where students in grades 3 and 5 were required to attend two 40-minute remedial sessions per week in which they first watched videos that reviewed the material that had been introduced in their math lessons that week and then played games to practice the skills introduced in the video. After four months, the intervention improved math achievement by 0.12 SDs. Many other evaluations of comparable interventions have found similar small-to-moderate results (see, e.g., Lai, Luo, Zhang, Huang, & Rozelle, 2015; Lai et al., 2012; Mo et al., 2015; Pitchford, 2015). These effects, however, have been consistently smaller than those of initiatives that adjust the difficulty of the material based on students’ performance (e.g., Banerjee et al., 2007; Muralidharan, et al., 2019). We hypothesize that these programs do little for learners who perform several grade levels behind curricular expectations, and who would benefit more from a review of foundational concepts from earlier grades.

We see two important limitations from this research. First, most initiatives that have been evaluated thus far combine instructional videos with practice exercises, so it is hard to know whether their effects are driven by the former or the latter. In fact, the program in China described above allowed learners to ask their peers whenever they did not understand a difficult concept, so it potentially also captured the effect of peer-to-peer collaboration. To our knowledge, no studies have addressed this gap in the evidence.

Second, most of these programs are implemented before or after school, so we cannot distinguish the effect of additional instructional time from that of the actual opportunity for practice. The importance of this question was first highlighted by Linden (2008), who compared two delivery mechanisms for game-based remedial math software for students in grades 2 and 3 in a network of schools run by a nonprofit organization in Gujarat, India: one in which students interacted with the software during the school day and another one in which students interacted with the software before or after school (in both cases, for three hours per day). After a year, the first version of the program had negatively impacted students’ math achievement by 0.57 SDs and the second one had a null effect. This study suggested that computer-assisted learning is a poor substitute for regular instruction when it is of high quality, as was the case in this well-functioning private network of schools.

In recent years, several studies have sought to remedy this shortcoming. Mo et al. (2014) were among the first to evaluate practice exercises delivered during the school day. They evaluated an initiative in Shaanxi, China in which students in grades 3 and 5 were required to interact with the software similar to the one in Lai et al. (2013) for two 40-minute sessions per week. The main limitation of this study, however, is that the program was delivered during regularly scheduled computer lessons, so it could not determine the impact of substituting regular math instruction. Similarly, Mo et al. (2020) evaluated a self-paced and a teacher-directed version of a similar program for English for grade 5 students in Qinghai, China. Yet, the key shortcoming of this study is that the teacher-directed version added several components that may also influence achievement, such as increased opportunities for teachers to provide students with personalized assistance when they struggled with the material. Ma, Fairlie, Loyalka, and Rozelle (2020) compared the effectiveness of additional time-delivered remedial instruction for students in grades 4 to 6 in Shaanxi, China through either computer-assisted software or using workbooks. This study indicates whether additional instructional time is more effective when using technology, but it does not address the question of whether school systems may improve the productivity of instructional time during the school day by substituting educator-led with computer-assisted instruction.

Increasing learner engagement

Another way in which technology may improve education is by increasing learners’ engagement with the material. In many school systems, regular “chalk and talk” instruction prioritizes time for educators’ exposition over opportunities for learners to ask clarifying questions and/or contribute to class discussions. This, combined with the fact that many developing-country classrooms include a very large number of learners (see, e.g., Angrist & Lavy, 1999; Duflo, Dupas, & Kremer, 2015), may partially explain why the majority of those students are several grade levels behind curricular expectations (e.g., Muralidharan, et al., 2019; Muralidharan & Zieleniak, 2014; Pritchett & Beatty, 2015). Technology could potentially address these challenges by: (a) using video tutorials for self-paced learning and (b) presenting exercises as games and/or gamifying practice.

Video tutorials

Technology can potentially increase learner effort and understanding of the material by finding new and more engaging ways to deliver it. Video tutorials designed for self-paced learning—as opposed to videos for whole class instruction, which we discuss under the category of “prerecorded lessons” above—can increase learner effort in multiple ways, including: allowing learners to focus on topics with which they need more help, letting them correct errors and misconceptions on their own, and making the material appealing through visual aids. They can increase understanding by breaking the material into smaller units and tackling common misconceptions.

In spite of the popularity of instructional videos, there is relatively little evidence on their effectiveness. Yet, two recent evaluations of different versions of the Khan Academy portal, which mainly relies on instructional videos, offer some insight into their impact. First, Ferman, Finamor, and Lima (2019) evaluated an initiative in 157 public primary and middle schools in five cities in Brazil in which the teachers of students in grades 5 and 9 were taken to the computer lab to learn math from the platform for 50 minutes per week. The authors found that, while the intervention slightly improved learners’ attitudes toward math, these changes did not translate into better performance in this subject. The authors hypothesized that this could be due to the reduction of teacher-led math instruction.

More recently, Büchel, Jakob, Kühnhanss, Steffen, and Brunetti (2020) evaluated an after-school, offline delivery of the Khan Academy portal in grades 3 through 6 in 302 primary schools in Morazán, El Salvador. Students in this study received 90 minutes per week of additional math instruction (effectively nearly doubling total math instruction per week) through teacher-led regular lessons, teacher-assisted Khan Academy lessons, or similar lessons assisted by technical supervisors with no content expertise. (Importantly, the first group provided differentiated instruction, which is not the norm in Salvadorian schools). All three groups outperformed both schools without any additional lessons and classrooms without additional lessons in the same schools as the program. The teacher-assisted Khan Academy lessons performed 0.24 SDs better, the supervisor-led lessons 0.22 SDs better, and the teacher-led regular lessons 0.15 SDs better, but the authors could not determine whether the effects across versions were different.

Together, these studies suggest that instructional videos work best when provided as a complement to, rather than as a substitute for, regular instruction. Yet, the main limitation of these studies is the multifaceted nature of the Khan Academy portal, which also includes other components found to positively improve learner achievement, such as differentiated instruction by students’ learning levels. While the software does not provide the type of personalization discussed above, learners are asked to take a placement test and, based on their score, educators assign them different work. Therefore, it is not clear from these studies whether the effects from Khan Academy are driven by its instructional videos or to the software’s ability to provide differentiated activities when combined with placement tests.

Games and gamification

Technology can also increase learner engagement by presenting exercises as games and/or by encouraging learner to play and compete with others (e.g., using leaderboards and rewards)—an approach known as “gamification.” Both approaches can increase learner motivation and effort by presenting learners with entertaining opportunities for practice and by leveraging peers as commitment devices.

There are very few studies on the effects of games and gamification in low- and middle-income countries. Recently, Araya, Arias Ortiz, Bottan, and Cristia (2019) evaluated an initiative in which grade 4 students in Santiago, Chile were required to participate in two 90-minute sessions per week during the school day with instructional math software featuring individual and group competitions (e.g., tracking each learner’s standing in his/her class and tournaments between sections). After nine months, the program led to improvements of 0.27 SDs in the national student assessment in math (it had no spillover effects on reading). However, it had mixed effects on non-academic outcomes. Specifically, the program increased learners’ willingness to use computers to learn math, but, at the same time, increased their anxiety toward math and negatively impacted learners’ willingness to collaborate with peers. Finally, given that one of the weekly sessions replaced regular math instruction and the other one represented additional math instructional time, it is not clear whether the academic effects of the program are driven by the software or the additional time devoted to learning math.

The prognosis:

How can school systems adopt interventions that match their needs.

Here are five specific and sequential guidelines for decisionmakers to realize the potential of education technology to accelerate student learning.

1. Take stock of how your current schools, educators, and learners are engaging with technology .

Carry out a short in-school survey to understand the current practices and potential barriers to adoption of technology (we have included suggested survey instruments in the Appendices); use this information in your decisionmaking process. For example, we learned from conversations with current and former ministers of education from various developing regions that a common limitation to technology use is regulations that hold school leaders accountable for damages to or losses of devices. Another common barrier is lack of access to electricity and Internet, or even the availability of sufficient outlets for charging devices in classrooms. Understanding basic infrastructure and regulatory limitations to the use of education technology is a first necessary step. But addressing these limitations will not guarantee that introducing or expanding technology use will accelerate learning. The next steps are thus necessary.

“In Africa, the biggest limit is connectivity. Fiber is expensive, and we don’t have it everywhere. The continent is creating a digital divide between cities, where there is fiber, and the rural areas.  The [Ghanaian] administration put in schools offline/online technologies with books, assessment tools, and open source materials. In deploying this, we are finding that again, teachers are unfamiliar with it. And existing policies prohibit students to bring their own tablets or cell phones. The easiest way to do it would have been to let everyone bring their own device. But policies are against it.” H.E. Matthew Prempeh, Minister of Education of Ghana, on the need to understand the local context.

2. Consider how the introduction of technology may affect the interactions among learners, educators, and content .

Our review of the evidence indicates that technology may accelerate student learning when it is used to scale up access to quality content, facilitate differentiated instruction, increase opportunities for practice, or when it increases learner engagement. For example, will adding electronic whiteboards to classrooms facilitate access to more quality content or differentiated instruction? Or will these expensive boards be used in the same way as the old chalkboards? Will providing one device (laptop or tablet) to each learner facilitate access to more and better content, or offer students more opportunities to practice and learn? Solely introducing technology in classrooms without additional changes is unlikely to lead to improved learning and may be quite costly. If you cannot clearly identify how the interactions among the three key components of the instructional core (educators, learners, and content) may change after the introduction of technology, then it is probably not a good idea to make the investment. See Appendix A for guidance on the types of questions to ask.

3. Once decisionmakers have a clear idea of how education technology can help accelerate student learning in a specific context, it is important to define clear objectives and goals and establish ways to regularly assess progress and make course corrections in a timely manner .

For instance, is the education technology expected to ensure that learners in early grades excel in foundational skills—basic literacy and numeracy—by age 10? If so, will the technology provide quality reading and math materials, ample opportunities to practice, and engaging materials such as videos or games? Will educators be empowered to use these materials in new ways? And how will progress be measured and adjusted?

4. How this kind of reform is approached can matter immensely for its success.

It is easy to nod to issues of “implementation,” but that needs to be more than rhetorical. Keep in mind that good use of education technology requires thinking about how it will affect learners, educators, and parents. After all, giving learners digital devices will make no difference if they get broken, are stolen, or go unused. Classroom technologies only matter if educators feel comfortable putting them to work. Since good technology is generally about complementing or amplifying what educators and learners already do, it is almost always a mistake to mandate programs from on high. It is vital that technology be adopted with the input of educators and families and with attention to how it will be used. If technology goes unused or if educators use it ineffectually, the results will disappoint—no matter the virtuosity of the technology. Indeed, unused education technology can be an unnecessary expenditure for cash-strapped education systems. This is why surveying context, listening to voices in the field, examining how technology is used, and planning for course correction is essential.

5. It is essential to communicate with a range of stakeholders, including educators, school leaders, parents, and learners .

Technology can feel alien in schools, confuse parents and (especially) older educators, or become an alluring distraction. Good communication can help address all of these risks. Taking care to listen to educators and families can help ensure that programs are informed by their needs and concerns. At the same time, deliberately and consistently explaining what technology is and is not supposed to do, how it can be most effectively used, and the ways in which it can make it more likely that programs work as intended. For instance, if teachers fear that technology is intended to reduce the need for educators, they will tend to be hostile; if they believe that it is intended to assist them in their work, they will be more receptive. Absent effective communication, it is easy for programs to “fail” not because of the technology but because of how it was used. In short, past experience in rolling out education programs indicates that it is as important to have a strong intervention design as it is to have a solid plan to socialize it among stakeholders.

Beyond reopening: A leapfrog moment to transform education?

On September 14, the Center for Universal Education (CUE) will host a webinar to discuss strategies, including around the effective use of education technology, for ensuring resilient schools in the long term and to launch a new education technology playbook “Realizing the promise: How can education technology improve learning for all?”

file-pdf Full Playbook – Realizing the promise: How can education technology improve learning for all? file-pdf References file-pdf Appendix A – Instruments to assess availability and use of technology file-pdf Appendix B – List of reviewed studies file-pdf Appendix C – How may technology affect interactions among students, teachers, and content?

About the Authors

Alejandro j. ganimian, emiliana vegas, frederick m. hess.

• Media Relations
• Terms and Conditions
• Privacy Policy

How technology is shaping learning in higher education

About the authors.

This article is a collaborative effort by Claudio Brasca, Charag Krishnan , Varun Marya , Katie Owen, Joshua Sirois, and Shyla Ziade, representing views from McKinsey’s Education Practice.

The COVID-19 pandemic forced a shift to remote learning overnight for most higher-education students, starting in the spring of 2020. To complement video lectures and engage students in the virtual classroom, educators adopted technologies that enabled more interactivity and hybrid models of online and in-person activities. These tools changed learning, teaching, and assessment in ways that may persist after the pandemic. Investors have taken note. Edtech start-ups raised record amounts of venture capital in 2020 and 2021, and market valuations for bigger players soared.

A study conducted by McKinsey in 2021 found that to engage most effectively with students, higher-education institutions can focus on eight dimensions  of the learning experience. In this article, we describe the findings of a study of the learning technologies that can enable aspects of several of those eight dimensions (see sidebar “Eight dimensions of the online learning experience”).

Eight dimensions of the online learning experience

Leading online higher-education institutions focus on eight key dimensions of the learning experience across three overarching principles.

Seamless journey

Clear education road map: “My online program provides a road map to achieve my life goals and helps me structure my day to day to achieve steady progress.”

Seamless connections: “I have one-click access to classes and learning resources in the virtual learning platform through my laptop or my phone.”

Engaging teaching approach

Range of learning formats: “My program offers a menu of engaging courses with both self-guided and real-time classes, and lots of interaction with instructors and peers.”

Captivating experiences: “I learn from the best professors and experts. My classes are high quality, with up-to-date content.”

Adaptive learning: “I access a personalized platform that helps me practice exercises and exams and gives immediate feedback without having to wait for the course teacher.”

Real-world skills application: “My online program helps me get hands-on practice using exciting virtual tools to solve real-world problems.”

Caring network

Timely support: “I am not alone in my learning journey and have adequate 24/7 support for academic and nonacademic issues.”

Strong community: “I feel part of an academic community and I’m able to make friends online.”

In November 2021, McKinsey surveyed 600 faculty members and 800 students from public and private nonprofit colleges and universities in the United States, including minority-serving institutions, about the use and impact of eight different classroom learning technologies (Exhibit 1). (For more on the learning technologies analyzed in this research, see sidebar “Descriptions of the eight learning technologies.”) To supplement the survey, we interviewed industry experts and higher-education professionals who make decisions about classroom technology use. We discovered which learning tools and approaches have seen the highest uptake, how students and educators view them, the barriers to higher adoption, how institutions have successfully adopted innovative technologies, and the notable impacts on learning (for details about our methodology, see sidebar “About the research”).

Double-digit growth in adoption and positive perceptions

Descriptions of the eight learning technologies.

• Classroom interactions: These are software platforms that allow students to ask questions, make comments, respond to polls, and attend breakout discussions in real time, among other features. They are downloadable and accessible from phones, computers, and tablets, relevant to all subject areas, and useful for remote and in-person learning.
• Classroom exercises: These platforms gamify learning with fun, low-stakes competitions, pose problems to solve during online classes, allow students to challenge peers to quizzes, and promote engagement with badges and awards. They are relevant to all subject areas.
• Connectivity and community building: A broad range of informal, opt-in tools, these allow students to engage with one another and instructors and participate in the learning community. They also include apps that give students 24/7 asynchronous access to lectures, expanded course materials, and notes with enhanced search and retrieval functionality.
• Group work: These tools let students collaborate in and out of class via breakout/study rooms, group preparation for exams and quizzes, and streamlined file sharing.
• Augmented reality/virtual reality (AR/VR): Interactive simulations immerse learners in course content, such as advanced lab simulations for hard sciences, medical simulations for nursing, and virtual exhibit tours for the liberal arts. AR can be offered with proprietary software on most mobile or laptop devices. VR requires special headsets, proprietary software, and adequate classroom space for simultaneous use.
• AI adaptive course delivery: Cloud-based, AI-powered software adapts course content to a student’s knowledge level and abilities. These are fully customizable by instructors and available in many subject areas, including business, humanities, and sciences.
• Machine learning–powered teaching assistants: Also known as chatbot programs, machine learning–powered teaching assistants answer student questions and explain course content outside of class. These can auto-create, deliver, and grade assignments and exams, saving instructors’ time; they are downloadable from mobile app stores and can be accessed on personal devices.
• Student progress monitoring: These tools let instructors monitor academic progress, content mastery, and engagement. Custom alerts and reports identify at-risk learners and help instructors tailor the content or their teaching style for greater effectiveness. This capability is often included with subscriptions to adaptive learning platforms.

Survey respondents reported a 19 percent average increase in overall use of these learning technologies since the start of the COVID-19 pandemic. Technologies that enable connectivity and community building, such as social media–inspired discussion platforms and virtual study groups, saw the biggest uptick in use—49 percent—followed by group work tools, which grew by 29 percent (Exhibit 2). These technologies likely fill the void left by the lack of in-person experiences more effectively than individual-focused learning tools such as augmented reality and virtual reality (AR/VR). Classroom interaction technologies such as real-time chatting, polling, and breakout room discussions were the most widely used tools before the pandemic and remain so; 67 percent of survey respondents said they currently use these tools in the classroom.

About the research

In November 2021, McKinsey surveyed 634 faculty members and 818 students from public, private, and minority-serving colleges and universities over a ten-day period. The survey included only students and faculty who had some remote- or online-learning experience with any of the eight featured technologies. Respondents were 63 percent female, 35 percent male, and 2 percent other gender identities; 69 percent White, 18 percent Black or African American, 8 percent Asian, and 4 percent other ethnicities; and represented every US region. The survey asked respondents about their:

• experiences with technology in the classroom pre-COVID-19;
• experiences with technology in the classroom since the start of the COVID-19 pandemic; and
• desire for future learning experiences in relation to technology.

The shift to more interactive and diverse learning models will likely continue. One industry expert told us, “The pandemic pushed the need for a new learning experience online. It recentered institutions to think about how they’ll teach moving forward and has brought synchronous and hybrid learning into focus.” Consequently, many US colleges and universities are actively investing to scale up their online and hybrid program offerings .

Differences in adoption by type of institution observed in the research

• Historically Black colleges and universities (HBCUs) and tribal colleges and universities made the most use of classroom interactions and group work tools (55 percent) and the least use of tools for monitoring student progress (15 percent).
• Private institutions used classroom interaction technologies (84 percent) more than public institutions (63 percent).
• Public institutions, often associated with larger student populations and course sizes, employed group work and connectivity and community-building tools more often than private institutions.
• The use of AI teaching-assistant technologies increased significantly more at public institutions (30 percent) than at private institutions (9 percent), though overall usage remained comparatively higher at private institutions.
• The use of tools for monitoring student progress increased by 14 percent at private institutions, versus no growth at public institutions.

Some technologies lag behind in adoption. Tools enabling student progress monitoring, AR/VR, machine learning–powered teaching assistants (TAs), AI adaptive course delivery, and classroom exercises are currently used by less than half of survey respondents. Anecdotal evidence suggests that technologies such as AR/VR require a substantial investment in equipment and may be difficult to use at scale in classes with high enrollment. Our survey also revealed utilization disparities based on size. Small public institutions use machine learning–powered TAs, AR/VR, and technologies for monitoring student progress at double or more the rates of medium and large public institutions, perhaps because smaller, specialized schools can make more targeted and cost-effective investments. We also found that medium and large public institutions made greater use of connectivity and community-building tools than small public institutions (57 to 59 percent compared with 45 percent, respectively). Although the uptake of AI-powered tools was slower, higher-education experts we interviewed predict their use will increase; they allow faculty to tailor courses to each student’s progress, reduce their workload, and improve student engagement at scale (see sidebar “Differences in adoption by type of institution observed in the research”).

While many colleges and universities are interested in using more technologies to support student learning, the top three barriers indicated are lack of awareness, inadequate deployment capabilities, and cost (Exhibit 3).

Students want entertaining and efficient tools

More than 60 percent of students said that all the classroom learning technologies they’ve used since COVID-19 began had improved their learning and grades (Exhibit 4). However, two technologies earned higher marks than the rest for boosting academic performance: 80 percent of students cited classroom exercises, and 71 percent cited machine learning–powered teaching assistants.

Although AR/VR is not yet widely used, 37 percent of students said they are “most excited” about its potential in the classroom. While 88 percent of students believe AR/VR will make learning more entertaining, just 5 percent said they think it will improve their ability to learn or master content (Exhibit 5). Industry experts confirmed that while there is significant enthusiasm for AR/VR, its ability to improve learning outcomes is uncertain. Some data look promising. For example, in a recent pilot study, 1 “Immersive biology in the Alien Zoo: A Dreamscape Learn software product,” Dreamscape Learn, accessed October 2021. students who used a VR tool to complete coursework for an introductory biology class improved their subject mastery by an average of two letter grades.

Faculty embrace new tools but would benefit from more technical support and training

Faculty gave learning tools even higher marks than students did, for ease of use, engagement, access to course resources, and instructor connectivity. They also expressed greater excitement than students did for the future use of technologies. For example, while more than 30 percent of students expressed excitement for AR/VR and classroom interactions, more than 60 percent of faculty were excited about those, as well as machine learning–powered teaching assistants and AI adaptive technology.

Eighty-one percent or more of faculty said they feel the eight learning technology tools are a good investment of time and effort relative to the value they provide (Exhibit 6). Expert interviews suggest that employing learning technologies can be a strain on faculty members, but those we surveyed said this strain is worthwhile.

While faculty surveyed were enthusiastic about new technologies, experts we interviewed stressed some underlying challenges. For example, digital-literacy gaps have been more pronounced since the pandemic because it forced the near-universal adoption of some technology solutions, deepening a divide that was unnoticed when adoption was sporadic. More tech-savvy instructors are comfortable with interaction-engagement-focused solutions, while staff who are less familiar with these tools prefer content display and delivery-focused technologies.

According to experts we interviewed, learning new tools and features can bring on general fatigue. An associate vice president of e-learning at one university told us that faculty there found designing and executing a pilot study of VR for a computer science class difficult. “It’s a completely new way of instruction. . . . I imagine that the faculty using it now will not use it again in the spring.” Technical support and training help. A chief academic officer of e-learning who oversaw the introduction of virtual simulations for nursing and radiography students said that faculty holdouts were permitted to opt out but not to delay the program. “We structured it in a ‘we’re doing this together’ way. People who didn’t want to do it left, but we got a lot of support from vendors and training, which made it easy to implement simulations.”

Reimagining higher education in the United States

Takeaways from our research.

Despite the growing pains of digitizing the classroom learning experience, faculty and students believe there is a lot more they can gain. Faculty members are optimistic about the benefits, and students expect learning to stay entertaining and efficient. While adoption levels saw double-digit growth during the pandemic, many classrooms have yet to experience all the technologies. For institutions considering the investment, or those that have already started, there are several takeaways to keep in mind.

• It’s important for administration leaders, IT, and faculty to agree on what they want to accomplish by using a particular learning technology. Case studies and expert interviews suggest institutions that seek alignment from all their stakeholders before implementing new technologies are more successful. Is the primary objective student engagement and motivation? Better academic performance? Faculty satisfaction and retention? Once objectives are set, IT staff and faculty can collaborate more effectively in choosing the best technology and initiating programs.
• Factor in student access to technology before deployment. As education technology use grows, the digital divide for students puts access to education at risk. While all the institution types we surveyed use learning technologies in the classroom, they do so to varying degrees. For example, 55 percent of respondents from historically Black colleges and universities and tribal colleges and universities use classroom interaction tools. This is lower than public institutions’ overall utilization rate of 64 percent and private institutions’ utilization rate of 84 percent. Similarly, 15 percent of respondents from historically Black colleges and universities and tribal colleges and universities use tools for monitoring student progress, while the overall utilization rate for both public and private institutions is 25 percent.
• High-quality support eases adoption for students and faculty. Institutions that have successfully deployed new learning technologies provided technical support and training for students and guidance for faculty on how to adapt their course content and delivery. For example, institutions could include self-service resources, standardize tools for adoption, or provide stipend opportunities for faculty who attend technical training courses. One chief academic officer told us, “The adoption of platforms at the individual faculty level can be very difficult. Ease of use is still very dependent upon your IT support representative and how they will go to bat to support you.”
• Agree on impact metrics and start measuring in advance of deployment. Higher-education institutions often don’t have the means to measure the impact of their investment in learning technologies, yet it’s essential for maximizing returns. Attributing student outcomes to a specific technology can be complex due to the number of variables involved in academic performance. However, prior to investing in learning technologies, the institution and its faculty members can align on a core set of metrics to quantify and measure their impact. One approach is to measure a broad set of success indicators, such as tool usage, user satisfaction, letter grades, and DFW rates (the percentage of students who receive a D, F, or Withdraw) each term. The success indicators can then be correlated by modality—online versus hybrid versus in-class—to determine the impact of specific tools. Some universities have offered faculty grants of up to $20,000 for running pilot programs that assess whether tools are achieving high-priority objectives. “If implemented properly, at the right place, and with the right buy-in, education technology solutions are absolutely valuable and have a clear ROI,” a senior vice president of academic affairs and chief technology officer told us.

In an earlier article , we looked at the broader changes in higher education that have been prompted by the pandemic. But perhaps none has advanced as quickly as the adoption of digital learning tools. Faculty and students see substantial benefits, and adoption rates are a long way from saturation, so we can expect uptake to continue. Institutions that want to know how they stand in learning tech adoption can measure their rates and benchmark them against the averages in this article and use those comparisons to help them decide where they want to catch up or get ahead.

Claudio Brasca is a partner in McKinsey’s Bay Area office, where Varun Marya is a senior partner; Charag Krishnan is a partner in the New Jersey office; Katie Owen is an associate partner in the St. Louis office, where Joshua Sirois is a consultant; and Shyla Ziade is a consultant in the Denver office.

The authors wish to thank Paul Kim, chief technology officer and associate dean at Stanford School of Education, and Ryan Golden for their contributions to this article.

Explore a career with us

Related articles.

Setting a new bar for online higher education

How to transform higher-education institutions for the long term

Scaling online education: Five lessons for colleges

• Open access
• Published: 13 May 2021

A meta-analysis on educational technology in English language teaching

• Jafar Rahmati 1 ,
• Siros Izadpanah   ORCID: orcid.org/0000-0002-2061-8110 1 &
• Ali Shahnavaz 2  

Language Testing in Asia volume  11 , Article number:  7 ( 2021 ) Cite this article

16k Accesses

8 Citations

6 Altmetric

Metrics details

As more various types of computer-assisted language learning (CALL) programs have been incorporated into language classrooms over the recent decades, it has become more important to uncover whether, to what extent, and under which moderator variables CALL can be yield more effective outcomes than traditional language instruction. The issue of education is one of the most important materials addressed by technology. Instead, meta-analysis is a statistical and quantitative method that leads us to a general conclusion by integrating the results of different researches. In this study, researchers worked on the impact of educational technology in English language teaching by studying 67 articles and theses (from 1000 studies that were relevant in title and abstract). All articles and theses were included from 2009 to 2020 and 7 articles were excluded from this study due to insufficient information. Furthermore, two instruments, SPSS (mainly its sub-branch Kruskal-Wallis test) and CMA were used to calculate and evaluate data in this research. The total effect size calculated for studies under both fixed and random models was statistically significant and also the study of effects by year of publication, instruments used in research and research methods showed that their effect size was significant. Teaching English with the help of technology has an effective effect size and has shown the success of this technology in language learning.

Introduction

Due to the rapid advances in Information and Communication Technology (ICT) in the world, there is growing attention to combine technologies into the classrooms to prepare learners to meet the needs of a progressively technological-dependent culture (Bond). The presence of technology and its constant advances have been disclosed into society by shifting the way how people cooperate with technology itself and through technology devices (Hollands & Escueta, 2020 ; Gonzalez-Acevedo, 2016 ). Warni, Aziz, and Febriawan ( 2018 ) believe that technology allows students to study independently and cooperate with their peers. This is possible because technology inspires students to reflect and analyze where these two capabilities are at the basics of developing autonomy. Since the 1960s, educational technologists have tried to make this image become a reality through emerging programs based on computer-assisted instruction (CAI) to drill, train, and test students (Andone & Frydenberg, 2019 ). According to Xiao ( 2019 ), every educator must utterly think about, update concepts, be courageous in innovation, let advanced science and technology assist college English education, and familiarize multimedia technology with a large amount of information, closeness and interactivity into college English teaching. Many educational researchers believe that computer-assisted language learning (CALL) would prove to be effective because it would decrease educational costs and increase learning outcomes in the long period (Atabek, 2020 ; Oz, Demirezen, & Pourfeiz, 2015 ).

Research in English language teaching sometimes contradicts differences in educational measures, situations, measurement instruments, and research methods that make it difficult for the researcher to easily compare the findings (Ozkale & Koc, 2020 ).

The disagreement between the results means that there is no acceptable answer to guide policymakers, and there is always an endless demand for re-research. There is a danger that the sponsors of social and educational research would conclude that this research is unproductive and unscientific. In addition, reviewing the sources of empirical research is usually not helpful, and because it depends so much on the opinion, judgment, preferences, and tendencies of the reviewers, conflicting interpretations of a piece of evidence are not uncommon.

However, examining the effectiveness of CALL is not easy for a number of reasons. First, the effect of any CALL program on learning outcomes is some way related to its uses. A specific CALL program may have great educational potential not revealed until it is used properly. Hence, evaluating the effectiveness of the CALL in language education is evaluating its uses rather than the CALL program itself. Second, the effectiveness of CALL is affected by some other moderator variables such as the learners, tasks, the educational setting conditions, and the assessment instruments. Third, CALL can be used either in isolation, as the sole instrument to convey language knowledge to the students, or in combination with traditional, face-to-face teaching methods (Sadeghi & Dousti, 2013 ). In addition to the above obstacles, no individual study by itself can show whether CALL programs are actually effective or not. In most countries, the use of educational technology is a headway and a national movement, and many organizations and educational institutions have been created in order to properly use educational technology and find better and more complete ways and techniques.

General objectives of the article: The role of educational technology in teaching English in Iran

Literature review

• Educational technology

An accurate understanding of the definition, subject, and scope of educational technology depends largely on the root meaning of the word technology. The word is derived from the word technology in Greek, meaning systematically performing art or profession. The first part of this word (technologia) is a combination of performing art and a technique involving knowledge of the meaning of the principles and the ability to achieve the desired results. In other words, logos mean practical things like knowing and doing. The word root means reasoning, explanation, principle, and ratiocination. So technology means the rational application of knowledge. The word consists of two words “technique” and “logic.” “Technique” means skillfully doing any work and “logic” is equivalent to the “knowing” suffix and means “science and knowledge.” Technology can, therefore, be regarded as methodology or knowledge and science in subtle ways of doing things. The second meaning is what the word “technology” is mostly used to express (Faradanesh, 2001 ).

Concept of educational technology

There are many definitions of educational technology, each referring to its various aspects. Before the application of technology in its new sense, planners helped improve the teaching and learning outcomes of audiovisual cases and devices. Thus, it can be concluded that the contributions of this branch are summarized from education to the use of purely educational items. But Brown ( 1972 ) has defined educational technology differently: Educational technology goes beyond the use of instruments. Educational technology is thus more than just a set of components (Ipek & Ziatdinov, 2018 ). It is a systematic way of designing, executing, and evaluating the whole process of research and learning to use specific goals, utilizing research findings in psychology and human communication, and employing a combination of human and non-human resources to create more effective learning, more reliable, and more deeply. In-depth attention to the above definition leads the reader to several basic conclusions:

The first conclusion to be drawn from the first part of the definition is that educational technology is not just about the use of educational instruments, but the broader scope of the use of educational instruments and the use of educational materials as part of it.

As educational technology is considered to be a systematic way or method, so it is more like an empire than its constituent parts because they are actions and reactions. Because the action, reaction, or interaction between its constituents lead to effects and results that are greater than the sum of its constituent elements.

Educational technology uses scientific findings such as psychology and the humanities to design and implement the whole process of teaching and learning.

Educational technology employs a good mix of human and non-human resources. In other words, unlike the use of educational materials or audio-visual training in which the use of material instruments is concerned, in technology, human resources are appropriately used.

The most recent definition agreed by educational technology experts, The American Association of Educational Communication & Technology (AECT) stated that educational technology is the theory and practice of designing, producing, using, and evaluating learning processes and resources (Spector & Yuen, 2016 ). Caffarela and Fly ( 1992 ) define this as considering that in any field of science philosophical issues such as epistemological epistemology and methodology are raised and that experts in different disciplines present theories in that field.

Application of educational technology in English language teaching

The application of educational technology in English language teaching includes any possible means and information that can be used in language teaching. It deals with language teaching instruments such as television, language labs, and a variety of designed media. In other words, the use of educational technology in language teaching is the same folk concept of educational technology as the use of audiovisual devices, monitors, and computer keyboards. The public domain of its audiovisual equipment consists of two distinct parts: the hardware and the software. The hardware talking part deals with physical and real equipment, such as projectors, sound recorders, TV sets, microcomputers, etc., and the software part includes many items used in connection with such equipment and devices like slide, audiotapes, videotapes of computer programs, written languages, and more (Ahdian, 2007 ); (Xu, Banerjee, Ramirez, Zhu, & Wijekumar, 2019 ).

Research in the field of education is sometimes contradictory. Differences in educational measures, situations, measuring tools, and research methods make it difficult for the researcher to compare the findings (Rai'i, Farzaneh, & Delavar, 2013 ). The contradiction between the results leads to no acceptable answer to be a guidance for policy makers (Talan, 2021 ). It also means that there is always an endless demand for research and re-research. There is a risk that research sponsors may conclude that this research is confusing, unproductive, and unscientific (Asgharpour, 2006 ).

Considering the research done in the field of language learning with the help of technology, it can be seen that there are a lot of disagreements about the success rate of teaching English with the help of technology. Some researchers such as Sung, Cheng and Liu ( 2016 ) and Lee ( 2010 ) are its defenders, while other researchers such as Lipsey and Wilson ( 2001 ), Norris and Ortega ( 2000 ), and Oswald and Plonsky ( 2010 ) have expressed doubts about its success.

Proponents of using technology for language learning, giving the learner freedom of action, access to a variety of language content, ease of access and its inherent attractiveness, opponents of factors such as lack of infrastructure, lack of software and hardware to especially in developing countries, students and teachers are not familiar with this technology, teachers do not master technology to produce appropriate educational content, superficial and unrealistic interactions in existing software, too much emphasis on the use of multimedia, heavy volume Content for language learners, lack of appropriate feedback and finally receiving superficial and unrealistic feedback as reasons for their opposition to using technology for language learning. These contradictory reasons led us to perform meta-analysis to determine the effectiveness of technology-assisted language learning and, in general, whether it was successful or not. These reasons became contradictory in order to determine the effectiveness of language learning through meta-analysis with the help of technology and to reach a general conclusion whether it was successful or not.

Research methods

Considering that the purpose of this research was to describe, analyze, and combine the studies presented in the field of educational technology in English language teaching based on research; the method of this research was a meta-analysis. Meta-analysis is a set of statistical methods that are performed in order to combine the results of independent experimental and correlation studies that have the same research questions on a single topic, and leads to a single estimate and result. Unlike traditional research methods, a meta-analysis uses statistical summaries of individual studies as research data.

Based on the main assumption of this method, each study provided different estimates of the underlying relationships in society. Therefore, by combining the results of these studies, a more accurate view of these relationships could be provided, which was provided by estimating individual studies. Meta-analysis research was applied type and was among the few pieces of research. The method used to collect data in this research was the library.

General objective subgroups: variables

Hypothesis 1 — There is a significant difference between the years of publication in research on the application of technology in English language teaching.

Hypothesis 2 — The research method used in the research has been used in the field of application of educational technology in English language teaching.

Hypothesis 3 — Research tools have been used in the application of educational technology in English language teaching.

Hypothesis 4 — There is a significant difference between the effect size of different statistical methods in research in the field of technology application in English language teaching.

Hypothesis 5 — There is a significant difference between the size of the work based on the gender of the sample in research on the application of technology through English language teaching.

Eligibility criteria and exclusion criteria

In this research, 67 articles or theses out of 1000 articles or theses which were related to technology in English language teaching in Iranian cites, like Tehran, Shahr-e- Qods, Yasouj, Shahrood, Mazandaran, Bandar Abbas, Alborz, Shahrekord, Ahvaz, Qeshm, Guilan, Semnan, and Chabahar, were randomly chosen from Iran Doc, google scholar, and science direct websites.

It is important to mention that 7 out of 67 articles and theses were ignored in this thesis based on the table that has been mentioned (Additional file 1 : Appendix A). Topics in selected articles were completely consistent and had a relatively high and appropriate subject similarity for meta-analysis were from the years between 2009 and 2020 (Additional file 1 : Appendix B). Conditions were detected and meta-analysis tests had been performed on them. It should be noted that in the meta-analysis method, there is no specific limit on the number of studies.

Method of data collection

To perform meta-analysis, the specifications of all theses in the field of educational technology in English language teaching, which are the year of publication, sampling method, statistical method, research method, and gender of the sample were studied. These data were then used in analysis, syntheses, and comparison.

Instruments

Meta-analysis is the statistical method which was used in this study. The SPSS (Sciences Statistical Package for the Social) software (SPSS Statistics 26) that researchers examined the frequency and statistical significance. The research hypotheses were also tested by SPSS software. CMA 2 software (Comprehensive Meta-Analysis version 2) was used to calculate the effect size for each study, the overall size effect, and the size of the discriminant effect to test their statistical significance in this research.

In addition, the research hypotheses were tested using SPSS software and the Kruskal-Wallis nonparametric test. The effect size in this r was calculated using the Hex method.

The Egger regression method has also been used to evaluate the homogeneity of the studies. The advantage of this method compared to other tests is that it is stronger. This method uses real effect size methods for prediction.

Data analysis

Part 1: descriptive analysis, description of general characteristics of the studied samples.

Descriptive information about the year of publication of the studies was examined in this study.

As can be seen in Table 1 , the highest percentage is related to research published during the years 2015 to 2017 with a rate of 51.7% and the lowest percentage is related to research published in the years 2018 to 2020 with a rate of 6.7 percentage.

Descriptive information about the research methods used in the studies reviewed in this study

As can be seen in Table 2 , the highest percentage is related to the quasi-experimental research method with 36.7% and the lowest percentage is related to the qualitative research method with 3.3%.

Descriptive information about the instruments used in the studies examined in this study

As can be seen in Table 3 , for the instruments used, the highest percentage is related to the Questionnaire instrument with 37% and the lowest percentage is related to the Observations instrument with a rate of 3.3%.

Descriptive information about the statistical method used in the studies examined in this study

As can be seen in Table 4 , the highest percentage is related to the method of using pre- and post-tests with 56.7% and the lowest percentage is related to the statistical method of ANOVA with 1.7%.

Descriptive information about the sex of the sample in the studies

As can be seen in Table 5 , for the sample gender, the highest percentage is related to mixed-gender with a rate of 48% and the lowest percentage is related to female gender with a rate of 20%.

Homogeneity of studies

In order to check the homogeneity of the studies, the Eger regression test is used and the results of this test are summarized in the following table:

As can be seen in Table 6 , due to the value of Sig, which is greater than 0.05, the assumption of homogeneity of studies at an error level of 0.05% is accepted.

The following Fig.  1 is used to determine whether the initial studies are biased and their impact on data analysis.

Funnel diagram shape the size of each study with the effect size accuracy

If the initial studies do not have a diffusion bias, they should be distributed symmetrically around the average effect size, as shown in the diagram above.

Overall effect size

Before examining the effect size separately for the variables in this study, the overall effect size is calculated in two modes: a model with random effects and a model with fixed effects, and the results are recorded in the table below.

It should be noted that due to the homogeneity of the initial studies in this study, the model with fixed effects is more efficient than the model with random effects.

As can be seen in Table 7 , considering that the sig value for both models is less than 0.01, it can be accepted that the total effect size in both models is significant with random effects and fixed effects at the error level of one percent.

Effect size by year of publication

The following table records the results related to the effect size by year of publication of studies in both model modes with random effects and fixed effects.

According to the Sig values obtained in Table 8 , the size of the effects in all the studied years is significant in both types of models with fixed effects and random effects.

The size of the work is used separately according to the research method

In the table below, the results related to the size of the effect are recorded separately by the research method used in the studies in both models with random effects and fixed effects.

According to the Sig values obtained in Table 9 , the size of the effects in all research methods used in the studies under study in both types of models with fixed effects and random effects are significant.

Effect size by the instrument used

In the table below, the results related to the size of the effect by instruments used in the studies are recorded in both model modes with random effects and with fixed effects.

According to the Sig values obtained in Table 10 , the size of the effects in all instruments used in the studies under study in both types of models with fixed effects and random effects are significant.

The size of the work is separated by statistical methods

In the table below, the results related to the effect size are recorded separately by the statistical methods used in the studies in both models with random effects and with fixed effects.

According to the Sig values obtained in Table 11 , the size of the effects in all statistical methods used in the studies in both types of models with fixed effects and random effects, except the random effects model in the case where the statistical instrument is used Qualitative Have been meaningful.

Effect size by sample gender

In the table below, the results related to the effect size by sample gender in both models with random effects and fixed effects are recorded.

According to the Sig values obtained in Table 12 , the size of the effects on the sex of the sample in both types of models with fixed effects and random effects are significant.

Part II: Inferential analysis

Hypothesis 1.

There is no significant difference between the size of the effect of years of the publication on research in the application of educational technology in English language teaching.

To test the above hypothesis, the Kruskal-Wallis test was used and the results of this test are recorded in the following tables:

As can be seen in Table 13 , considering the value of Sig = 0.151, which is greater than 0.05, the assumption of zero, i.e. the assumption that the size of the work is the same according to the year of publication is not rejected at the level of 5% error.

The effect of years of publication in research in the field of technology application in English language teaching is not significantly different.

Hypothesis 2

There is no significant difference between the size of the work and the research method used. Research conducted in the field of technology application in English language teaching.

To test the above hypothesis, the Kruskal-Wallis test was used and the results of this test are recorded in the following table:

As can be seen in Table 14 , considering the value of Sig = 0.302, which is greater than 0.05, the null hypothesis, i.e. the assumption that the size of the work is the same, is not rejected at the 5% error level, so researchers can say: There is no significant difference between the size of the work and the research method used in the field of technology application in English language teaching.

Hypothesis 3

There is no significant difference between the size of the work and the instruments used in research in the field of technology application in English language teaching.

To test the above hypothesis, Kruskal-Wallis test was used and the results of this test are recorded in the following table:

As can be seen in Table 15 , considering the value of Sig = 0.830, which is greater than 0.05, the null hypothesis, i.e. the assumption that the size of the work is the same, is not rejected at the 5% error level, so researchers can say: There is no significant difference between the size of the work and the instruments used in research in the field of technology application in English language teaching.

Hypothesis 4

There is no significant difference between the size of the work according to the statistical method used in research in the field of technology application in English language teaching.

To test the above hypothesis, Kruskal-Wallis test was used and the results of this test are recorded in the following tables:

As can be seen in Table 16 , considering the value of Sig = 0.814, which is greater than 0.05, the null hypothesis, i.e. the assumption that the size of the work is the same, is not rejected at the 5% error level, so researchers can say: There is no significant difference between the size of the work according to the statistical method used in research in the field of technology application in English language teaching.

Hypothesis 5

There is no significant difference between the size of the work by gender of the sample in research in the field of technology application through English language teaching.

As can be seen in Table 17 , considering the value of Sig = 0.819, which is greater than 0.05, the null hypothesis, i.e. the assumption which the size of the work is the same, is not rejected at the 5% error level, so researchers can say: There is no significant difference between the size of the work by gender of the sample in research in the field of technology application through English language teaching.

In this part, researchers describe the collected results in general and discuss the statistical results obtained. The present study includes 67 studies out of 1000 theses and articles which 7 of them were excluded from this study due to a lack of sufficient information ( Appendix A ).

The main purpose of this study was to investigate the impact of educational technology on English language teaching. The optimal research method to achieve this goal was meta-analysis. In this method, “each research” was a unit of study, furthermore, the amount of effect size was calculated for each research in order to obtain the effectiveness of each research.

Our results indicate that technology applications have a large effect (1.68 and 0.91, fixed effect model and random effect model respectively) on English language teaching. This proposes that the use of technology is more effective than traditional teaching methods without technology for English language teaching quality.

Overall effects of educational technology on English language teaching

The result of a medium-sized overall positive effect of educational technology on English language teaching confirmed that the use of a computer, telegram, mobile, laptop devices, and software could facilitate language learning. These results were consistent with other research findings regarding the effects of different devices and software on English language teaching.

Related to the first research question: Year of publication

This research question was in line with Sung, Yang, and Lee ( 2017 ) and Chauhan ( 2017 ), which both had the same experimental results show that their meta-analysis was not substantially affected by publication bias. The most obvious finding to emerge from this research question was that years of publication did not have a significant result in this research.

Based on the fact, the year of publication was selected for research as a variable; if years are considered differently, that is, for example, the year 2009 is assumed alone, they are meaningful. They also have the same feature for the year of publication until 2020. However, based on the research question of how much the effect of the year of publication affects educational technology, it should be noted that this variable is not recommended for future research because it changes every time based on advances in technology and different methods for research. Considering the year of publication, it will not have a significant effect as a whole on the effect size of the work.

Related to the second research question: research method

It is in line with Farzaneh Shakki ( 2015 ). There is no significant difference between the effect size by the research method used and the research conducted in the field of technology application in the English language teaching.

In fact, the research method as a whole depends on the researcher and the type of research that is being done. In this research, we conclude that if we want to consider the research methods one by one we can claim that they all have a significant effect but when we want to consider all of them relative to each other, they do not have a significant effect. Therefore, this research shows us that the required research methods or resources required as well as different goals can be variable, so it depends on the researcher in what circumstances, in what environment and with what tools they can choose the research method. Of course, a single research method may not be used in an article, and several types can be used.

Related to the third research question: instruments

It was in line with Fazeli ( 2016 ). It was not in line with Pourtayebi ( 2015 ), Alinejad ( 2015 ), Sadeqi ( 2015 ), Rastegar ( 2014 ), Shahkooei ( 2016 ), and Parinaz ( 2010 ). There is no significant difference between the effect size and the instruments used in research in the field of technology application in English language teaching

As we have seen, among the number of theses and articles we reviewed, a variety of instruments were used. In the meantime, the questionnaire was used more than other instruments, but this does not mean that this instrument is superior to others in research instruments. In this study, in each of the articles and theses, one or more instruments were used, which were significant, but in general, they were not significant in comparison to each other. This means that we cannot say which tool is better than other instruments so it depends on the researcher which instrument to choose over the research.

Related to the fourth research question: statistical method

It was not in line with Shahkooei ( 2016 ). There is no significant difference between the size of the work according to the statistical method used in research in the field of technology application in English language teaching. Although the number of statistical methods used in these studies was different, in the ranking, they did not have a significant difference.

To check the quantitative research data, the use of statistical tests is mandatory. A statistical method is necessary to use for each research. Reviewing all statistical tests can be a good guide for analyzing data in an article. Meanwhile, it may not always be enough to use a test.

Statistical methods are one of the practical ways to identify problems and provide solutions to managerial, social, and psychological problems, etc. that, if implemented correctly, can provide real data for our research.

In other words, there may be different ways of doing research or how we can collect our data to prove or answer questions. At this point, having high analytical power, problem-solving ability. And sufficient experience can help you to know the correct method of research. Because it directly uses people’s opinions, it can solve society’s problems, and these studies are often very practical and can be cited.

Related to the fifth research question: gender

It was in line with Alipour ( 2017 ), Sadeqi ( 2015 ), Nateghi ( 2018 ), Dayani ( 2014 ), Ghazavi ( 2017 ), and Aliakbari ( 2013 ). It was not in line with Mohammadi ( 2014 ), Alashti ( 2013 ), AsgharHeidari ( 2014 ), and Nakhaei ( 2017 ) found the result of the study that the attitudes of English teachers or students regarding their gender towards the use of the Internet, mobile or other devices were positive and high.

According to the statistical part of this study, the participants were mixed in most of the articles, but in some of them, exclusively female participants or in some other male participants were used to conduct the research. Based on the findings, we conclude that there is not much difference between men being superior to women or vice versa.

Based on the availability of technology in education, educational technology has caused many changes in the field that meet the needs of students in different ways. With the provision of software that teaches students with special needs, the appropriate educational equipment is designed to make learning easier for the individual.

With the use of technology, the concept of education is changing for both students and teachers to progress. Therefore, the introduction of technology in education is very important.

Research limitations

The present meta-analysis, like many others, has its limitations and forces the researcher to interpret the findings with extreme caution.

Lack of access to some articles and dissertations that did not receive a response from the authors despite sending an email.

Suggestion for further study

Due to the limitations that researchers applied in this research, 67 theses and articles were selected from different cities of Iran that had a topic related to the subject of this research, but it should be noted that due to development and progress in recent years, the importance of this thesis is observed. It is better to select researches that have been published in reputable publications all over the world, in addition to this, it is suggested to work on other various variables.

A meta-analysis of research on the application of technology in English language teaching, which was published in valid journals in this field and examined, showed that the application of information technology in this field has an acceptable impact factor.

In this section, the overall findings of the current study were presented. According to the studied variables, we conclude that the five variables studied and researched, according to their statistical information in this study, did not have a significant effect size. And in response to the overall purpose of the article, how much technology can affect English language teaching, it can be concluded that, initially, compared to the variable of the year, 2017 to 2020, the size of the work was more representative than previous years, so technology has been effective. Other variables, such as tools, research methods, statistical methods, and gender, have had a smaller effect than size that we can ignore.

The results showed that all chosen variables in this study, considering every unique thesis or article, were significant, but as the whole consideration of each variable to 60 theses and articles, they were not significant.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Information and communication technology

Computer-assisted language learning

Abbassi, Z. (2015). A comparative study of applying websites and mobile phones on learning collocations among Iranian EFL learners . Thesis (Unpublished). Khatam University.

Ahdian, M. (2007). Introduction to Educational Technology, Tehran, Iran .

Alashti, L. A. (2013). Exploring and comparing the attitudes of Iranian English teachers and learners toward using the Internet for language learning . Thesis (Unpublished). Semnan University.

Aliakbari, M., (2013). Teachers’ perception of the barriers to critical thinking. Procedia-Social and Behavioral Sciences, 70 , 1–5. https://doi.org/10.1016/j.sbspro.2013.01.031 .

Alinejad, J. (2015). The impact of interactive whiteboard on developing vocabulary learning , Thesis (Unpublished) (). Shahrood: Islamic Azad University.

Google Scholar  

Alipour, M. (2017). The effect of using social networks on developing EFL learners’ pragmatic competence through speaking accuracy and fluency . Thesis (Unpublished). Islamic Azad University.

Andone, D., & Frydenberg, M. (2019). Creating virtual reality in a business and technology educational context. In Augmented reality and virtual reality (pp. 147-159). Springer.

Ardebili, M. (2013). Computer-assisted language learning: attitudes of Iranian EFL learners . Thesis (Unpublished). Urmia University.

AsgharHeidari, F. (2014). The impact of media analysis on the development of critical thinking and reading comprehension skills of the EFL learners . Thesis (Unpublished). University of Guilan.

Asgharpour, D. (2006). Bibliography of research methods and methodology of humanities. Seminary and University of Humanities Methodology , 12 (48), 154–167.

Atabek, O. (2020). Associations between emotional states, self-efficacy for and attitude towards using educational technology. International Journal of Progressive Education , 16 (2), 175–194. https://doi.org/10.29329/ijpe.2020.241.12 .

Article   Google Scholar  

Bozorgmanesh, B. (2013). Innovative versus traditional approaches to vocabulary teaching: examining effects of online quotations in teaching vocabulary in Iranian EFL context . Thesis (Unpublished). Ilam University.

Brown, J. W. (1972). Administering educational media: Instructional Technology and Library Services . USA: McGraw-Hill Companies.

Caffarella, E. P., & Fly, K. (1992). Developing a knowledge base and taxonomy in instructional technology . Retrieved April 22, 2021 from https://www.learntechlib.org/p/144856/.

Chauhan, S. (2017). A meta-analysis of the impact of technology on learning effectiveness of elementary students. Computers & Education , 105 , 14–30. https://doi.org/10.1016/j.compedu.2016.11.005 .

Dayani, A. (2014). The effect of using blogs on Iranian EFL learners’ critical thinking . thesis (unpublished). Shahid Chamran University.

Dousti, M. (2012). The effect of length of exposure to call technology on young Iranian EFL learners’ grammar and vocabulary gain . Thesis (Unpublished). Urmia University.

Ebrahimi, H. (2014). The impact of teaching- medium on EFL learners’ meta-pragmatic information: the case of compliment . Thesis (Unpublished). University of Guilan.

EbrahimiSeraji, N. (2016). Teacher’s attitudes toward educational technology in language institutes in Mazandaran . Thesis (Unpublished). Islamic Azad University.

Esferjani, R. (2017). Effect of internet-assisted pronunciation instruction on Iranian l2 learners’ production of primary stress in English compound nouns. Thesis (Unpublished). Shahrekord University, Shahrekord.

Esmaili, S. (2012). EFL writing and computer-assisted language learning: towards an “eclectic” view and far “beyond” that . Thesis (Unpublished). Chabahar Maritime University.

Ezzatian, S. (2013). The differential effects of debate and media analysis as relate to enhancing EFL learners’ critical thinking ability and writing performances . Thesis (Unpublished). Sheykhbahayee University.

Faradanesh, H (2001). Theoretical foundations of educational technology . Tehran: Organization for the Study and Compilation of Humanities Books.

Fard, M. K. (2016). A comparative study on the use of teaching aids in Iranian EFL institutes and schools . Thesis (Unpublished). Payame Noor University.

Farhesh, S. (2012). On the effect of using PPT slides on vocabulary and grammar acquisition, and students’ attitudes . Thesis (Unpublished). Allameh Tabataba’i University.

Farshadnia, S. (2010). Culture in online communication: netiquette applications and implementations in TEFL community . Thesis (Unpublished). Al-Zahra University.

Fazeli, S. A. (2016). An investigation into the effectiveness of call in the writing ability of young adolescent EFL learners: with some reference to the perceptions of call teachers and learners . Thesis (Unpublished). Yasouj University.

Ghazavi, A. Z. (2017). EFL student’s assumptions on using smartphone applications in learning English language: a case study of students at the Department of English, Razi University . Thesis (Unpublished). Razi University.

Ghaziyani, N. N. (2017). The effect of selective feedback on paragraph writing performance of Iranian intermediate EFL learners in telegram as a social network setting . Thesis (Unpublished). Payame Noor University.

Gonzalez-Acevedo, N. (2016). Technology-enhanced-gadgets in the teaching of English as a foreign language to very young learners. Ideas on implementation. Procedia-Social and Behavioral Sciences, 232 , 507-513. https://doi.org/10.1016/j.sbspro.2016.10.070 .

Hassanzadeh, M. (2010). A comparison and contrast of two distance education systems (e-learning vs. print-based learning) in the reading comprehension skill . Thesis (Unpublished). Payam-e-Noor University.

Hollands, F., & Escueta, M. (2020). How research informs educational technology decision-making in higher education: the role of external research versus internal research. Educational Technology Research and Development , 68 (1), 163–180. https://doi.org/10.1007/s11423-019-09678-z .

Inanloo, F. (2017). The impact of community-based web sites, online discussions forums and chat rooms on l2 learners’ expressive writing skill improvement . Thesis (Unpublished). Islamic Azad University.

Ipek, I., & Ziatdinov, R. (2018). New approaches and trends in the philosophy of educational technology for learning and teaching environments. arXiv preprint arXiv.

Javdani, M. (2017). Iranian EFL teachers’ attitudes toward blended learning approach . Thesis (Unpublished). Islamic Azad University.

Kashani, N. N. (2015). The impact of multimedia-supported metacognitive listening strategies instruction on Iranian EFL learners’ listening proficiency . Thesis (Unpublished). Shahid Rajaee Teacher Training University.

KhademianHashemi, H. (2014). The effect of video-mediated listening texts on EFL learners’ writing ability . Thesis (Unpublished). Islamic Azad University.

Khalilabad, M. H. (2016). The effect of multimedia texts presented on the interactive whiteboard (smart board) on high school EFL learners’ reading comprehension . Thesis (Unpublished). Hakim Sabzevari University.

Khalili, S. (2013). Vocabulary instruction through blended learning and multimedia softwares in Iranian EFL classes . Thesis (Unpublished). Sheikhbahaee University, Esfahan.

Khazaee, S. (2017). Comparing the efficacy of chatting as one of the call-based activities on children vs. adult EFL learners’ speaking ability . Thesis (Unpublished). Chabahar Maritime University.

Lee, I. (2010). Writing teacher education and teacher learning: testimonies of four EFL teachers. Journal of Second Language Writing, 19 (3), 143–157. https://doi.org/10.1016/j.jslw.2010.05.001.

Letafati, M. (2013). The effect of asynchronous forum discussions on the writing ability of Iranian EFL learners . Thesis (Unpublished). Payam-e-Noor University.

Li, S. (2010). The effectiveness of corrective feedback in SLA: a meta-analysis. Language Learning , 61 (2), 319–365.

Lipsey, M. W. ,& Wilson, D. B. (2001). Practical meta-analysis . Thousand Oaks: Sage.

Mardian, F. (2014). A socio-cultural study of the impact of computer-mediated corrective feedback on the development of EFL learners’ grammatical knowledge . Thesis (Unpublished). Alzahra University.

Mohammadi, M. (2014). The relationship between internet use and academic procrastination of EFL learners . Thesis (Unpublished). University of Guilan.

Najafi, S. (2013). The effects of call multimedia (still-picture and motion-picture multimedia) on the vocabulary learning of elementary EFL students . Thesis (Unpublished). Shahid Madani University.

Nakhaei, M. (2017). The effect of using telegram as a social network on vocabulary learning of iranian students . Thesis (Unpublished). Islamic Azad University.

Nami, F. (2020). Educational smartphone apps for language learning in higher education: students’ choices and perceptions. Australasian Journal of Educational Technology , 5 , 82–95.

Nateghi, M. (2018). The effect of mobile learning for collaborative dialogues on EFL Iranian learners’ speaking performance . Thesis (Unpublished). Islamic Azad University.

Noghani, F A. (2015). The effect of multimodal presentation on EFL learners’ listening comprehension and self-efficacy . Thesis (Unpublished). Hakim Sabzevari University.

Noori, L. (2014). The effect of wiki on complexity, accuracy and fluency of intermediate EFL learners’ writing performance . Thesis (Unpublished). University of Zanjan.

Norris, J. M., & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and a quantitative meta-analysis. Language Learning, 50, 417–528.

Oswald, F. L., & Plonsky, L. (2010). Meta-analysis in second language research: choices and challenges. Annual Review of Applied Linguistics , 31 , 85.

Oz, H., Demirezen, M., & Pourfeiz, J. (2015). Digital device ownership, computer literacy, and attitudes toward foreign and computer-assisted language learning. Procedia-Social and Behavioral Sciences , 186 , 359–366. https://doi.org/10.1016/j.sbspro.2015.04.028 .

Ozkale, A., & Koc, M. (2020). Investigating academicians’ use of tablet PC from the perspectives of human computer interaction and Technology Acceptance Model. International Journal of Technology in Education and Science , 4 (1), 37–52. https://doi.org/10.46328/ijtes.v4i1.36 .

Parinaz, I. (2010). The impact of web-based reading lessons on EFL students reading comprehension, motivation, and autonomy . Thesis (Unpublished). Al-Zahra University.

Parisa, P. (2017). On the relationship between media literacy and personality traits and Iranian EFL learners’ listening comprehension . Thesis (Unpublished). Imam Khomeini International University.

Paslar, Z. (2017). Politeness strategies used by intermediate Iranian EFL learners in online what’s app interactions . Thesis (Unpublished). Islamic Azad University.

Poorkhalil, M. R. (2016). The effect of using multimedia on vocabulary learning of basic and elementary Iranian EFL learners . Thesis (Unpublished). Semnan University.

Pour, A. M. (2015). The impact of synchronous computer-mediated communication on writing ability, patterns of collaboration and motivation of Iranian EFL learners . Thesis (Unpublished). University of Guilan.

Pourtayebi, H. (2015). The effect of mobile instant messaging on reading comprehension ability . Thesis (Unpublished). Hakim Sabzevari University.

Rafei, A. (2017). The relationship between willingness to communicate and internet use among Iranian EFL learners . Thesis (Unpublished). Shahid Rajaee Teacher Training University.

Rai'i, Farzaneh, & Delavar, A. (2013). Meta-analysis: the art of correcting the mistakes of others. Journal of Educational Measurement , 4 (13), 119–132.

Rajabi, S. (2016). Traditional class environment (TCE) vs. mobile assisted language learning (mall): the case of intermediate Iranian EFL learners English idioms achievement. Thesis (Unpublished). Alborz University of Higher Education.

Rastegar, S. (2014). The effect of interactive text-messaging on learning English phrasal verbs by Iranian EFL learners . Thesis (Unpublished). University of Mazandaran.

Raye-Ahmadi, M. V. (2014). Media literacy in the mediation of English language learning . Ph.D. DISSERTATION. Alzahra University.

Razavi, A. (2016). Constructing and validating a multimedia techniques (MTS) scale and examining the impact of using technology in teaching English in Iranian high schools on students’ attitudes, anxiety, and language proficiency . Thesis (Unpublished). Imam Reza International University.

Sadeqi, M. (2015). The effects of call and online resources on learning collocation of Iranian EFL learners . Thesis (Unpublished). Islamic Azad University.

Sadeghi, K., & Dousti, M. (2013). The effect of length of exposure to CALL technology on young Iranian EFL learners' grammar gain. English Language Teaching, 6 (2), 14–26. https://doi.org/10.5539/elt.v6n2p14 .

Salimi, H. (2016). The application of wiki-mediated collaborative writing as a pedagogical instrument to promote ESP learners’ writing performance, autonomy, and perceptions of uses of wikis . Thesis (Unpublished). Allameh Tabataba’i University.

Shafaghiha, G. (2016). Watching English language TV series and visual literacy of Iranian EFL learners: challenges, possibilities, reasons . Thesis (Unpublished). Alzahra University.

Shafati, M. (2013). The effects of etymological elaboration and multimedia instruction on EFL learners’ idiom retention . Thesis (Unpublished). Sheikhbahaee University.

Shahkooei, S. (2016). A comparison of traditional flash-card vs. mobile learning: Using mms for learning idioms by Iranian EFL learners (idiomobile) . Thesis (Unpublished). Allameh Mohaddes Nouri University.

Shakki, F. (2015). The effect of dynamic assessment on EFL learners’ listening comprehension through mediational strategies . Thesis (Unpublished). Islamic Azad University.

Shargh, S. N. (2010). The pedagogical effectiveness of multimedia call software on vocabulary recall and retention of Iranian pre-intermediate EFL learners . Thesis (Unpublished). Ferdowsi University.

Spector, J. M., & Yuen, A. H. (2016). Educational technology program and project evaluation . Routledge. https://doi.org/10.4324/9781315724140 .

Book   Google Scholar  

Sung, Y. T., Chang, K. E., & Liu, T. C. (2016). The effects of integrating mobile devices with teaching and learning on students’ learning performance: A meta analysis and research synthesis. Computers & Education, 94 , 252–275. https://doi.org/10.1016/j.compedu.2015.11.008 .

Sung, Y. T., Yang, J. M., & Lee, H. Y. (2017). The effects of mobile-computer-supported collaborative learning: meta-analysis and critical synthesis. Review of Educational Research , 87 (4), 768–805. https://doi.org/10.3102/0034654317704307 .

Talan, T. (2021). The effect of simulation technique on academic achievement: a meta-analysis study. International Journal of Technology in Education and Science, 5 (1), 17–36. https://doi.org/10.14686/buefad.706231 .

Warni, S., Aziz, T. A., & Febriawan, D. (2018). The use of technology in English as a foreign language learning outside the classroom: an insight into learner autonomy. LLT Journal: A Journal on Language and Language Teaching , 21 (2), 148–156.

Xiao, L. (2019). Application development of modern multimedia technology in English teaching. Frontiers in Educational Research , 2 (2), 12–39.

Xu, Z., Banerjee, M., Ramirez, G., Zhu, G., & Wijekumar, K. (2019). The effectiveness of educational technology applications on adult English language learners’ writing quality: a meta-analysis. Computer Assisted Language Learning , 32 (1-2), 132–162. https://doi.org/10.1080/09588221.2018.1501069 .

Zakeri, D. (2018). A study of the effects of using multimedia in teaching technical translation model for Iranian students . Thesis (Unpublished). Islamic Azad University.

Zanussi, M. P. (2015). The impact of watching captioned TV series on vocabulary development of EFL students . Thesis (Unpublished). Allameh Mohaddes Nouri University.

Zarat-ehsan, M. (2015). The impact of multimedia glosses on intermediate EFL learners’ vocabulary learning and retention . Thesis (Unpublished). University of Guilan.

Download references

Author information

Authors and affiliations.

Department of English Language Teaching, Zanjan Branch, Islamic Azad University, Zanjan, Iran

Jafar Rahmati & Siros Izadpanah

Department of Mathematics and Statistics, Zanjan Branch, Islamic Azad University, Zanjan, Iran

Ali Shahnavaz

You can also search for this author in PubMed   Google Scholar

Contributions

All authors of the research had more or less equal contributions in the process of conception, design, acquisition of data, analysis, and interpretation of data. They have all been involved in revising the manuscript critically to the same extent. All take public responsibility for the whole content. All are equally accountable for all aspects of the work. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Siros Izadpanah .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1. .

Abbassi, 2015 , Ardebili, 2013 , Bozorgmanesh, 2013 , Dousti, 2012 , Ebrahimi, 2014 , EbrahimiSeraji, 2016 , Esferjani, 2017 , Esmaili, 2012 , Ezzatian, 2013 , Fard, 2016 , Farhesh, 2012 , Farshadnia, 2010 , Ghaziyani, 2017 , Hassanzadeh, 2010 , Inanloo, 2017 , Javdani, 2017 , Kashani, 2015 , KhademianHashemi, 2014 , Khalilabad, 2016 , Khalili, 2013 , Khazaee, 2017 , Letafati, 2013 , Li, 2010 , Mardian, 2014 , Najafi, 2013 , Nami, 2020 , Noghani, 2015 , Noori, 2014 , Parisa, 2017 , Paslar, 2017 , Poorkhalil, 2016 , Pour, 2015 , Rafei, 2017 , Rajabi, 2016 , Raye-Ahmadi, 2014 , Razavi, 2016 , Salimi, 2016 , Shafaghiha, 2016 , Shafati, 2013 , Shargh, 2010 , Zakeri, 2018 , Zanussi, 2015 , Zarat-ehsan, 2015 , Zhang & Liu, 2019.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Rahmati, J., Izadpanah, S. & Shahnavaz, A. A meta-analysis on educational technology in English language teaching. Lang Test Asia 11 , 7 (2021). https://doi.org/10.1186/s40468-021-00121-w

Download citation

Received : 26 December 2020

Accepted : 13 April 2021

Published : 13 May 2021

DOI : https://doi.org/10.1186/s40468-021-00121-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• English language teaching
• Meta-analysis

• Information Systems
• Information Systems (Business Informatics)

Learning and Teaching with Technology: Effectiveness of ICT integration in schools

• February 2021
• Thesis for: BSc

• Ambo University

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations
• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

• People and community

UCLA’s Joe Way on leading the charge to transform learning technology

Share this article.

By Madeline Adamo, UCLA

On paper, Joe Way’s job at UCLA might sound like he’s implementing wall-sized transparent computer screens and holographic telepresence on campus. And if that technology were readily available today, there’s no doubt he would be bringing it to Westwood. 

Until then, UCLA’s executive director of digital spaces is working to make technology more consistent across campus and planning future digital infrastructure to benefit the Bruin community. Think video systems in residence halls — similar to those in hotels — that learn users’ habits and needs or digital wayfinding signage that better integrates emergency messaging on campus. 

Way, who joined UCLA almost a year ago, has plenty of experience in educational technology. His current role in UCLA IT Services was created to help lead the  Digital Campus Roadmap  — an initiative providing the pathway for an information technology support model that helps advance  goal 4  of the UCLA’s Strategic Plan: to elevate teaching in a way that centers on inclusive excellence.  

“Four walls don’t make a classroom: Learning can happen anywhere,” said Way, who in 2019 formed a higher education technology managers alliance that now has over 6,000 members and represents 1,400 institutions worldwide. “Students think more of a college experience, not just going to class. That’s only 10% of their time.” 

Newsroom spoke with Way about how he and his teams are elevating campus technologies and what he’s seeing in the latest learning trends.  

This interview has been edited for length and clarity.

What do you do at UCLA? 

I oversee classroom and live event technologies pretty much across the enterprise. That includes our general use classrooms — of which there are around 200— as well as our administrative conference rooms and live events at spaces like the (UCLA Meyer and Renee) Luskin Conference Center and the Hill. Looking ahead, I’ll be reimagining our digital signage.

Through the pandemic, a lot of services we didn’t think of beforehand were exposed, like the ability to record classes. Even now, back on campus, recording gives students an opportunity to worry less about taking notes. Being able to have transcription and translation services also really helps for digital access and equity. 

I was brought in to see how we can bring these services to all of our classrooms. It’s really looking at how we can make them effective for the students and faculty, with more active learning that uses classroom time for discussion and application of the principles that might have been watched online ahead of time.

What is a project you’re working on, and why is it important? 

In the subcommittee on modernizing the classroom, there are 10 of us looking at the current state of our classrooms. We’re also looking at the trends in higher education to inform how we can make our classrooms more effective for new ways of teaching and learning. After this phase, we’ll be putting together our comprehensive strategic plan that my group then will be then implementing.

We’re going to look at social spaces and how we can integrate those into the residence halls. We’ve seen a big move toward group studies. If we look at the research library, there’s ergonomics to allow that collaboration. You have to be able to turn chairs around and push tables together easily. So, we’re really looking at this holistically. I think taking what we naturally want to do as humans and bringing that into teaching and learning is going to be how we can make it more effective.

For example, as more TVs are installed in residence halls, why not also integrate a hotel system that will allow students to flip on the TV in their room and scroll down a menu to watch their class from the day before? We could send messaging through it; they could order their Starbucks through it because the system would start to understand them. You probably get in your car and it knows where you want to go because it gets used to your habits. Why not bring that into our campus experience?

What do you love about working at UCLA?

There’s a feeling here of community that I’ve never felt before. To be able to meet former Chancellor Gene Block, to see now Interim Chancellor Darnell Hunt, and to see and work with Vice Chancellor Michael Beck … I feel it from the top down — the sense of people rooting for you and saying, “I want you to be successful; we have an opportunity to do this.” 

That environment encourages me to keep wanting to do this work. It doesn’t matter who I’m talking to or what department. I don’t feel like it’s siloed at all. There’s a support system, and you feel that people are proud of what we’re about to embark on. I think having this strategic plan and having Chancellor-designate Julio Frenk coming in January will really give us that opportunity to have an impact that will outlive all of us. 

Keep Reading

• UC Spotlight: September 2024 September 16, 2024 Our monthly UC Spotlight celebrates the UC locations, teams and individual staff members who are helping to make UC a great place to work.
• Adelí Durón marks the fifth anniversary of the UC Irvine Latinx Resource Center September 16, 2024 Since its creation, the Laxinx Resource Center withstood the challenges of the pandemic and emerged as one of the finest centers of its kind in the U.S.
• La UC celebra el Mes Nacional de la Herencia Hispana con cifras récord de estudiantes latinos matriculados September 13, 2024  Para honrar el Mes Nacional de la Herencia Hispana, la Universidad de California celebra los logros y las contribuciones de los latinos que han sido …
• Employee News
• Public notices archive (through 2023)

 / 

THESIS PROPOSAL: Further development of Autonomous Gokart Platform

Ansök senast: 2024-12-01

Autonomous Platform, Electronics, Autonomous Drive, Machine Learning, Embedded Development, Embedded Systems  

TARGETED PROGRAMS

BSc or MSc Electrical Engineering, Automation & Mechatronics Engineering , Automotive Engineering    

PURPOSE OF THE STUDY  

To increase the knowledge and expertise within the evolving automotive industry, Infotiv Technology Development AB developed the open and educational Autonomous Platform.  It is currently in its fourth iteration (AP4) and the goal of the platform is to have an autonomous gokart that can be used to progress research and development in the automotive industry. This thesis aims to continue the development of a selected technical area of AP4. The following topics are suggested:  

• Exploring autonomous vehicle software stacks such as Autoware, Nvidia Drive, Openpilot, etc.  
• Localization for efficient navigation  
• Obstacle detection and avoidance   
• Adding new functionalities and sensors (sensor fusion)  
• Improving the verification and validation process  
• Build HIL/SIL testing facilities  

Potential research questions:  

• Enhancing localization for autonomous navigation  
• Real time systems in centralized E/E Architecture  
• Verification and validation of complex embedded system or autonomous vehicle software stack  

Learn more about the autonomous platform project in this GitHub repository:   https://github.com/infotiv-research/autonomous_platform  

WHO ARE WE LOOKING FOR?  

We are looking for 2 master students with a background in mechatronics , electrical engineering, or equivalent program , who wish to conduct their thesis during the spring of 202 5 . Applicants shall have experience in electrical engineering , software development and an interest in learning more about test automation. Applicants shall also have experience from programming languages such as C++ or Python, and it is meritorious to have previous experience from or knowledge of Continuous Integration, Delivery or Deployment, Jenkins, Git , Test Automation, ROS2 and Machine learning .   

TechDev is a department at Infotiv who focuses on SW & HW development and test solutions. We currently consist of 70 technical consultants with diverse backgrounds and experience from many technological fields. Our employees use their expertise to provide tailored solutions to all kinds of challenges, ranging from SW development, machine learning and simulations, to project & test management and way of working. One of our key strengths is the friendly atmosphere in our technical community, which provides access to TechDev's collective knowledge through internal collaboration tools and competence leader program, continuously providing updates in the latest tech.  

HOW TO APPLY  

Apply for this thesis no later than 2024-12-01. Assure to attach your resumé and a short summary of why you want to partake in this thesis.  We only consider applicants avaliable for on-site work at our office in Gothenburg. 

For further information, contact: Maria Alemyr [[email protected]] +46(0) 76 890 78 72 .

Fler jobb från Infotiv :

Liknande jobb:

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 16 September 2024

Deep behavioural representation learning reveals risk profiles for malignant ventricular arrhythmias

• Maarten Z. H. Kolk   ORCID: orcid.org/0000-0003-0624-1510 1 , 2 ,
• Diana My Frodi   ORCID: orcid.org/0000-0001-7074-5575 3 ,
• Joss Langford   ORCID: orcid.org/0000-0002-3204-5866 4 , 5 ,
• Tariq O. Andersen 6 ,
• Peter Karl Jacobsen   ORCID: orcid.org/0000-0002-1520-8774 3 ,
• Niels Risum 3 ,
• Hanno L. Tan 1 , 7 ,
• Jesper Hastrup Svendsen   ORCID: orcid.org/0000-0001-8466-8515 3 , 8 ,
• Reinoud E. Knops 1 , 2 ,
• Søren Zöga Diederichsen   ORCID: orcid.org/0000-0002-9687-0857 3 &
• Fleur V. Y. Tjong   ORCID: orcid.org/0000-0001-9608-0081 1 , 2  

npj Digital Medicine volume  7 , Article number:  250 ( 2024 ) Cite this article

Metrics details

• Arrhythmias
• Cardiovascular diseases

We aimed to identify and characterise behavioural profiles in patients at high risk of SCD, by using deep representation learning of day-to-day behavioural recordings. We present a pipeline that employed unsupervised clustering on low-dimensional representations of behavioural time-series data learned by a convolutional residual variational neural network (ResNet-VAE). Data from the prospective, observational SafeHeart study conducted at two large tertiary university centers in the Netherlands and Denmark were used. Patients received an implantable cardioverter-defibrillator (ICD) between May 2021 and September 2022 and wore wearable devices using accelerometer technology during 180 consecutive days. A total of 272 patients (mean age of 63.1 ± 10.2 years, 81% male) were eligible with a total sampling of 37,478 days of behavioural data (138 ± 47 days per patient). Deep representation learning identified five distinct behavioural profiles: Cluster A ( n  = 46) had very low physical activity levels and a disturbed sleep pattern. Cluster B ( n  = 70) had high activity levels, mainly at light-to-moderate intensity. Cluster C ( n  = 63) exhibited a high-intensity activity profile. Cluster D ( n  = 51) showed above-average sleep efficiency. Cluster E ( n  = 42) had frequent waking episodes and poor sleep. Annual risks of malignant ventricular arrhythmias ranged from 30.4% in Cluster A to 9.8% and 9.5% for Clusters D-E, respectively. Compared to low-risk profiles (D-E), Cluster A demonstrated a three-to-four fold increased risk of malignant ventricular arrhythmias adjusted for clinical covariates (adjusted HR 3.63, 95% CI 1.54–8.53, p  < 0.001). These behavioural profiles may guide more personalised approaches to ventricular arrhythmia and SCD prevention.

Similar content being viewed by others

Prediction of atrial fibrillation from at-home single-lead ECG signals without arrhythmias

Prediction of mortality from 12-lead electrocardiogram voltage data using a deep neural network

Detecting cardiac pathologies via machine learning on heart-rate variability time series and related markers

Introduction.

Malignant ventricular arrhythmias are a main cause of sudden cardiac death (SCD) 1 . In individuals at increased risk of SCD, patterns in physical behaviour (such as physical activity levels, sedentary behaviours, sleep behaviour) have emerged as potential prognostic indicators for ventricular arrhythmia onset, heart failure progression, and patient-reported outcomes 2 , 3 , 4 , 5 . Wearable accelerometers provide a means for continuous measurement of these day-to-day physical behaviours in free-living environments 6 . Identifying patterns or clusters within behavioural time-series data requires dimensionality reduction, as traditional clustering algorithms are unable to effectively process the granularity of such complex datasets. The process of dimensionality reduction, for instance reducing data to summary measures, may lead to the loss of intricate, non-linear associations in the data. Alternatively, deep neural networks are capable of learning low-dimensional latent representations from these complex datasets, while preserving the richness and intrinsic information present in the data 7 , 8 . Unsupervised machine learning algorithms can then operate on these latent space representations to categorise similar samples into one cluster 9 , 10 .

In this study, we aimed to identify and characterise behavioural profiles through deep representation learning in patients at risk of malignant ventricular arrhythmias (Fig. 1 ). Patients with an implantable cardioverter-defibrillator (ICD) were followed for six consecutive months using a wearable accelerometer to continuously monitor physical behaviour. A deep neural network was trained to learn a compressed representation from the behavioural time-series data while preserving the relevant information. We hypothesised that through the clustering of these deep behavioural representations, we would be able to identify clinically meaningful behavioural profiles. These profiles were evaluated for their clinical relevance and association with the risk of ventricular arrhythmia.

The workflow of the study is illustrated, that includes recruitment, data collection through a wearable device and data processing to identify behavioural profiles. a Recruitment of 303 patients from two international sites, who wore a wearable accelerometer for 180 consecutive days, during which behavioural metrics were recorded. b A convolutional residual variational autoencoder learned the latent behavioural representations. The models was provided with time-series data for each subject that consisted of 27 variables measured at 180 timepoints. c Unsupervised clustering using a k-means algorithm of the behavioural representations identified distinct behavioural profiles.

A total of 303 participants were enroled in SafeHeart, of which 272 met the eligibility criteria for this study (21 patients did not wear the GENEActiv wearable accelerometer, 10 patients did not meet the required minimum of 30 days of behavioural data). A total of 37,478 days of wearable data were collected (mean 138 ± 47 days per patient). Table 1 shows the clinical characteristics of the patient cohort. Patients had a mean age of 63.1 ± 10.4 years and 80.9% were male, 133 (48.9%) patients had ischaemic heart disease as cause of heart failure, 147 (54.0%) had heart failure with reduced ejection fraction (HFrEF), and 187 (68.8%) had a secondary prevention ICD indication. Fifty (18.4%) patients received cardiac resynchronisation therapy (CRT), the majority of patients used a β-blocker (80.5%). All patients completed one-year of follow-up, during which 46 (16.9%) patients received appropriate ICD therapy for a malignant ventricular arrhythmia, five (1.8%) patients received inappropriate ICD therapy and four (1.4%) patients died.

Characteristics of the identified behavioural profiles

Five behavioural profiles were identified: A ( n  = 46), B ( n  = 70), C ( n  = 63), D ( n  = 51) and E ( n  = 42). Mean values for behavioural metrics across the clusters are displayed in Table 2 , Supplementary Fig. 4 provides a granular representation of behaviours during the 180-day monitoring period. Figure 2 shows the individual behavioural metrics for each profile, relative to the cohort averages. In summary, Clusters B and C were characterised by active profiles with high daily steps counts (15353 ± 4062 & 13577 ± 4130 steps). The volume of activity (2921 ± 1410 g s) in Cluster B was accumulated over a longer period (370 ± 76 min) and at a lower average intensity (123 ± 17 m g ), while the volume of activity (2527 ± 810 g s) in Cluster C was linked to a greater number of faster walking steps (3689 ± 2100 steps) at higher cadences (92 ± 10 steps/min) and intensities (132 ± 22 g s). Clusters D and E had less active profiles with fewer daily steps (9971 ± 3103 & 9291 ± 3378 steps). Cluster D had the longest inactive bout durations and fewest number per day (0.72 ± 0.25 mins and 632 ± 110). The behavioural patterns of Cluster E were more fragmented with shorter active bouts (0.43 ± 0.12 min) and more inactive bouts (863 ± 135). Cluster A had the highest inactive duration (835 ± 110 min), the lowest activity intensity (103 ± 33 m g ) and least number of steps (6246 ± 2406 steps). In the sleep domain, Cluster A was characterised by the shortest total sleep duration (281 ± 86 min), the longest average duration of wake after sleep onset interruptions (6.8 ± 2.1 min) and the longest sleep onset latency (10.8 ± 5.1 min). The nocturnal patterns of Cluster E were fragmented, similar to the day, with the lowest sleep efficiency (51.5 ± 8.6%), most wake after sleep onset interruptions (32 ± 8) and shortest maximum sleep bout lengths (42 ± 10 min). Cluster D had the long sleep interval and total sleep durations (611 ± 122 and 369 ± 75 min) with longest maximum sleep bout lengths (57 ± 11 min). The sleep profiles of Clusters B and C were unexceptional other than Cluster B having the shortest sleep interval duration (507 ± 89 min).

The average values for the behavioural measurements for each behavioural profile are displayed. a The bar charts displayed on the left depict the average values for the metrics that reflect movement behaviour across behavioural profiles, relative to the cohort average. b Bar charts depicted on the right display the average values for the metrics that reflect sleep behaviours across the behavioural profiles. All values were scaled using z-scores.

Cluster characterisation

SHAP values were computed to represent the most important behavioural markers that characterised each behavioural profile. Figure 3a shows the variables with highest feature importance across behavioural profiles. The duration spent in moderate activity, the amount of slow steps and the number of sleep events were the behavioural markers that differentiated most between clusters. Figure 3b–f illustrates the top behavioural features that predict membership of each of the clusters. Clusters C and E were predicted by a combination of sleep behaviours and movement behaviours, while the other clusters were predominantly predicted by movement behaviours alone. No statistically significant differences were observed between clusters in terms of medication usage and medical history, apart from hypertension ( p  = 0.037) (Table 1 ).

A trained machine learning classifier (extreme gradient boosting) was used to predict membership of a profile based on daily behavioural measurements. a The bar chart represents the importance of features used by the extreme gradient boosting model to predict each profile. Horizontal bars represent the average contribution of a behavioural metric for the predicted profile. The features are ranked based on the summed importance of that feature to predict each profile. b – f The SHAP summary plot is displayed for each behavioural profile. The features are ranked by the mean absolute SHAP value. A positive SHAP value suggests a positive contribution, while a negative value indicates a negative contribution. The model predicted membership of the behavioural profile based on the daily measurements with an AUROC of 0.99.

Patient-reported outcomes across behavioural profiles

A total of 239 patients filled out questionnaires at the study baseline (non-response rates for subsequent clusters A-E were 11.4%, 15.6%, 12.7%, 9.5%, and 15.2%, respectively). Median scores for the EQ-5D-5L and KCCQ domains are provided in Supplementary Table 2 . In particular, Cluster A reported physical limitations, Cluster C reported high self-efficacy but worse social limitations, Cluster D highest disease-specific quality of life, and Cluster E reported highest burden of symptoms (Supplementary Fig. 5 ). Differences in patient-reported outcomes between clusters were not statistically significant.

Incidence of the outcomes of interest across behavioural profile

Figure 4a shows the risk of malignant ventricular arrhythmias treated by the ICD across the clusters during one-year follow-up. Event rates for clusters A until E were respectively 30.4%, 17.1%, 17.5%, 9.8% and 9.5% (log-rank p value 0.06). As displayed in Fig. 4b , the risk of malignant ventricular arrhythmias was significantly higher in Cluster A (unadjusted HR 2.26, 95% CI 1.20–4.23, p  = 0.01), which remained after adjusting for clinical covariates (adjusted HR 2.30, 95% CI 1.21–4.36, p  = 0.01). Also, the risk of malignant ventricular arrhythmias in the low-risk behavioural profiles (Cluster D-E) was significantly lower compared to the other clusters (unadjusted HR 0.45, 95% CI 0.22–0.94, p  = 0.03). Inappropriate ICD therapy was delivered in three patients in Cluster A (4.3%), two patients in Cluster B (2.9%), and one patient in Cluster C (1.6%). In total four patients died during follow-up, of which two in Cluster D (3.9%), one in Cluster A (2.2%) and one in Cluster C (1.6%). A significant difference in the composite endpoint between clusters was observed (log-rank p value 0.04) (Fig. 4c ). Unadjusted and adjusted hazard ratios for the respective clusters for the composite endpoint are displayed in Table 3 . In Supplementary Fig. 6 , ROC curves of logistic regression models predicting cases of malignant ventricular arrhythmias and the composite endpoint are presented. Regression models that included cluster membership within their feature set demonstrated superior performance, compared to models that excluded this variable.

Time-to-event analyses according to the behavioural profiles are presented. a Kaplan-Meier curves for malignant ventricular arrhythmias treated by the ICD, and b hazard ratios and 95% confidence intervals obtained from the Cox proportional-hazards model. c Kaplan-Meier curves for the composite endpoint of all ICD therapy and mortality, and d hazard ratios and 95% confidence intervals. The prevalence of the outcome is displayed as a percentage, represented by the blue and green circles. Distributions of times to events were compared with the log-rank test.

In this study, we demonstrated deep representation learning of complex day-to-day movement and sleep behaviours to enable the identification of clinically relevant behavioural profiles. These profiles were associated with an annual risk of malignant ventricular arrhythmias ranging from 30.4% to 9.5%. Our research extends prior work, bringing forth two novelties. First, while prior studies have evaluated physical behavioural metrics over monitoring intervals up to 14 days 3 , we identified distinct behavioural profiles derived from continuous accelerometer measurements spanning six months. Second, earlier studies have mainly focused on individual metrics for activity or sleep, despite these 24-hour rest-activity behaviours being highly interrelated. In the present work, we took a more holistic approach to physical behaviour by modelling the interplay between various concurrent behavioural mechanisms and their potential implications for clinical events.

Despite considerable variations in clinical trajectories among patients with an ICD, current follow-up strategies remain one-size-fits-all . Advances in wearable technologies have removed barriers for the continuous measurement of behavioural patterns, which could make wearables suitable as screening tools to identify individuals at-risk of disease progression. Several studies have shown that continuous activity measurements could indicate a decline in functional status, progression of heart failure, or onset of atrial fibrillation, each potentially increasing the risk of ventricular arrhythmia onset 3 , 4 , 11 , 12 . However, physical activity is also a modifiable risk factors that may reduce ventricular arrhythmia risk by alteration of autonomic tone, mitigation of the catecholamine release observed during exercise and an increase of resting parasympathetic tone 13 . Recent analyses of data from the UK Biobank have demonstrated a reduction in the risk of ventricular arrhythmia amongst physically active individuals 14 , 15 . With data from this prospective study, we demonstrated that an active behavioural profile does not necessarily reduce the risk of ventricular arrhythmia, which highlights the importance of considering various behaviours simultaneously. In particular, Clusters B and C had annual event rates of ~17% despite their daily time spent physically active being substantially higher compared to the other profiles. In contrast, Clusters D and E were half as likely to experience the outcome of interest, despite having less active profiles. This indeed suggests that interplays between various behaviours, such as intermittent sedentary behaviour with isolated bouts of physical exertion, rather than isolated measurements of activity characteristics, may explain differences in risk of ventricular arrhythmia onset. Furthermore, the absence of significant associations between patient-reported outcomes (e.g. symptom severity and physical limitations) and behavioural profiles might indicate that these are phenotypic in their origin rather than representing more transient behavioural patterns.

Our findings support the notion that abnormalities in 24-hour rest-activity patterns modulate the risk of ventricular arrhythmia onset. Circadian rhythm disruption has been associated with increased risk of atrial fibrillation onset 16 and heart failure 17 in previous studies. We observed an annual risk of ventricular arrhythmia exceeding 30% in the behavioural profile characterised by sedentary behaviour, a lack of high-intensity activity, and disturbed sleep behaviour. Adjusted for clinical covariates, this profile was associated with a three-to-four fold risk of experiencing a ventricular arrhythmia compared to the low-risk profiles. While these findings should be validated in larger cohorts, they emphasise the importance of comprehensive modelling of physical behaviour. The use of wearable devices for behavioural profiling holds promise for follow-up strategies tailored to an individual patient.

Clustering of deep learning-derived latent representations comes with the limitation of interpretability, as the latent representations are inferred from the underlying data and are not directly explainable ( black box ). To provide transparency, we characterised clusters by assessment of feature importance of a trained machine learning classifier that predicts cluster memberships based on day-to-day behavioural metrics 18 . A second limitation to our study is the use of processed output from the accelerometer, instead of the underlying raw accelerometry output. Some of these metrics are created through application of specific thresholds that rely on the calibration studies, but pose a challenge when comparing metrics among different studies or populations 19 . Third, from our findings, it remains uncertain whether the behavioural profiles can be generalised to other populations, such as heart failure patients who do not satisfy the criteria for an ICD, and thus warrant future research. Fourth, despite cluster membership showing significant associations with the outcome of interest after adjusting for clinical covariates, there is a risk of residual confounding. For instance, high-risk behavioural profile (Cluster A) was characterised by higher proportion of patients with a prior myocardial infarctions; however, these patients did not receive prescriptions for β-blockers, lipid-lowering drugs, or ACE inhibitors. This could point towards a potential undertreatment of these patients. This study was entirely decentralised in its design, with patient recruitment, informed consent, and study procedures conducted without physical contact between the study staff and participants. Consequently, information from imaging modalities (e.g., LVEF) and electrocardiography at the time of enrolment was not available. Future studies exploring the interplay between these clinical patient characteristics, and behavioural profiles are warranted.

Deep representation learning of physical behavioural patterns identifies distinct behavioural profiles with significant differences in their risk of malignant ventricular arrhythmia and death. Behavioural profiling using objective and real-time measurements obtained from wearable devices may enable clinicians to adjust and optimise treatment and prevention strategies to an individual patient. Interpretability of clustered latent representations and relatively small sample sizes prompt the need for further investigation into the mechanisms underlying their influence on ventricular arrhythmia risk and SCD.

The study was approved by the Institutional Review Boards of the Amsterdam University Medical Center (date 09-04-2021, approval number 2020/248) and Copenhagen University Hospital Rigshospitalet (date 19-04-2021, approval number H-20081068). All participants provided informed consent prior to their enrolment. The study was conducted in accordance with the Declaration of Helsinki.

Study design and setting

This is an analysis of the international SafeHeart study, a prospective, observational study conducted at two tertiary academic centers in Europe (Amsterdam University Medical Center, the Netherlands and Copenhagen University Hospital Rigshospitalet, Denmark). The purpose of this study was to develop a personalised model to predict ICD therapy for malignant ventricular arrhythmia 20 . Data used to create the prediction model included recordings from a wearable accelerometry recording device. Patient inclusion was conducted through telephone-based procedures between May 2021 and September 2022, the enrolment date was defined as the day when the wearable device was delivered to the patient. Throughout the study, participants had the option to withdraw from the study at any stage, either partially (by discontinuing the use of the wearable device) or completely. We adhered to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines for observational studies 21 . The study was registered at the National Trial Registration in the Netherlands (Trial NL9218; https://www.onderzoekmetmensen.nl/en ).

Participants

Participants qualified for enrolment in the SafeHeart study if they fulfilled the following conditions: i) received an ICD with or without cardiac resynchronisation therapy (CRT-D) in the five years leading up to enrolment, ii) experienced appropriate or inappropriate ICD therapy (high voltage shock therapy or anti-tachycardia pacing (ATP)) or demonstrated evidence of ventricular arrhythmias within eight years prior to enrolment, iii) engaged in a remote ICD monitoring programme, and iv) were at least 18 years old. Exclusion criteria were severe physical disability, end-stage heart failure, and a life expectancy of less than one year. The study protocol with the entire list of inclusion and exclusion criteria has been published previously 20 .

Physical behaviour measurements

Accelerometer-based wearable devices allow for continuous and objective quantification of daily physical behaviour by the recording of body movement along reference axes and signal analysis (e.g., intensity, frequency, and volume of activity and postural changes). In this study, various behavioural metrics were collected including daily activity and inactivity durations, duration of activity and inactivity episodes, activity intensity, activity volume, step count (total, slow and fast), cadence, sleep duration, sleep efficiency, wake up after sleep onset (WASO), nap duration and sleep onset latency. A complete overview of the collected metrics and their definitions are displayed in Supplementary Table 1 . To collect these metrics, participants wore the GENEActiv Original 1.1 accelerometer (Activinsights Ltd, Cambridgeshire, United Kingdom) on the wrist for 6 months. Devices were returned (for data extraction) and replaced biweekly or every 4 weeks. Continuous raw data were recorded at 50 Hz or 20 Hz and converted into daily summaries 22 , 23 . Patients were eligible for this study if they had at least 30 days of wearable data.

Outcome of interest

The prospective collection of the outcomes of interest occurred at both sites from enrolment in the study onwards. These outcomes were: i) any malignant ventricular arrhythmia defined as an episode of sustained ventricular tachycardia or ventricular fibrillation, treated by the ICD through a shock and/or ATP; ii) a composite endpoint comprising all ICD therapies and death. ICD therapies encompassed those for malignant ventricular arrhythmias, in addition to those in response to rhythms other than sustained ventricular tachycardia or ventricular fibrillation (e.g. atrial fibrillation, sinus tachycardia).

Patient reported outcome measures

Two patient reported outcome measures (PROMs), the EuroQoL 5-Dimensions 5-Levels (EQ-5D-5L) and Kansas City Cardiomyopathy Questionnaire (KCCQ), were used in the SafeHeart study 24 , 25 . Both PROMs were filled out by participants at study enrolment. The EQ-5D-5L assesses health across five domains, yielding a utility score ranging from −0.590 to 1.000. Meanwhile, the KCCQ, designed for heart failure patients, provided scores on a scale of 0 to 100, subdivided into the domains symptom burden, physical limitation, social limitation, quality of life and self-efficacy.

Deep representation learning of physical behaviour data

We derived deep representations from the day-to-day behavioural time-series collected during the first six months of the study (Fig. 1a ). Specifically, we used a β-variational autoencoder (VAE) that encodes input data through a probabilistic approach (mapping data into a probability distribution) and decodes from this distribution back into reconstructed data (Fig. 1b ) 26 , 27 . Supplementary Fig. 1 presents a schematic overview of the VAE architecture. The inputs were longitudinal trajectories of 27 behavioural metrics over 180 days, resulting in an input dimension of 272 × 27 × 180. Missing values of behavioural metrics were linearly interpolated and normalised. Our trained VAE reconstructed the behavioural time-series with a Pearson Correlation Coefficient of 0.988 ± 0.0379, a root mean square error (RMSE) of 0.031 ± 0.026 and a percentage root-mean-square difference (PRD) of 10.553 ± 0.038. Supplementary Fig. 2 depicts an example of the trends in behavioural measurements along with the reconstructed trend derived from 32 latent variables. The VAE models were developed using PyTorch (version 2.0.5) in Python (version 3.6.7).

We then applied an unsupervised machine learning algorithm to cluster these representations (Fig. 1c ). The k-means algorithm aims to minimise the within-cluster variance, making data points within the same cluster as similar as possible and data points in different clusters as dissimilar as possible. The appropriate number of clusters was assessed using within-cluster variation (inertia), silhouette scores, and Davies-Bouldin index (Supplementary Fig. 3 ). Considering that the k-means algorithm operates stochastically, and initialisation of the model may affect the decision of the optimal k , we averaged the results over multiple iterations to reduce the impact of randomness 28 . We evaluated cluster stability by computing the Jaccard index across 100 bootstrapped samples 29 . Clustering was performed using the scitkit-learn library (version 1.3.0) 30 .

Cluster characterisation through cluster membership prediction

We aimed to characterise the identified clusters using SHapley Additive exPlanations (SHAP) values 31 . SHAP values are widely used to determine the contribution of particular features to the predicted outcome. We derived SHAP values from a supervised machine learning classifier (eXtreme Gradient Boosting), which was trained to predict cluster membership from daily behavioural values (48,960 days) 32 . Subsequent ranking of these SHAP values provides insight into behavioural metrics that contribute positively (or negatively) to cluster membership. We assessed the performance of these classifications using the receiver operating characteristic curve (ROC).

Statistical analysis

Continuous variables were presented by the median, mean, interquartile range, and standard deviation. Categorical socio-demographic and clinical variables were presented as frequencies (percentages) and compared using the χ 2 test. T-tests were used for pairwise comparisons, analysis of variance (ANOVA) for assessing differences among multiple groups with normally distributed data. The Mann–Whitney U test was used for non-normally distributed variables, and the Kruskal-Wallis test for comparisons involving more than two groups with non-normally distributed data. The risk of the outcomes of interest during follow-up was estimated using the Kaplan–Meier method; log-rank tests were used to compare survival between clusters. Cox Proportional Hazard models were used to assess the association between behavioural profiles and the risk of outcomes of interest. The model included the clinical covariates age, sex, indication for ICD implantation, presence of atrial fibrillation, heart failure, and type of ICD. Schoenfeld residuals were used to check the proportional hazards assumption. A two-sided p value < 0.05 was considered significant. The prognostic significance of the behavioural profiles for the outcomes of interest was assessed through logistic regression models. Two models were constructed for each outcome of interest: the first model included clinical patient information (medical history and medication status) along with cluster membership as input features, while the second model excluded cluster membership. Prediction accuracy was assessed through stratified k-fold cross-validation, and quantified using the area under the receiver operating characteristic curve (AUROC).

Data availability

Data sharing requests will be considered upon a reasonable request. For access, please email the corresponding author.

Code availability

Code scripts are available at: https://github.com/DeepRiskAUMC/Deep-representation-clustering .

Empana, J. P. et al. Incidence of sudden cardiac death in the European Union. J. Am. Coll. Cardiol. 79 , 1818–1827 (2022).

Article   PubMed   Google Scholar  

Shim, J., Fleisch, E. & Barata, F. Wearable-based accelerometer activity profile as digital biomarker of inflammation, biological age, and mortality using hierarchical clustering analysis in NHANES 2011-2014. Sci. Rep. 13 , 9326 (2023).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Kolk, M. Z. H. et al. Accelerometer-assessed physical behavior and the association with clinical outcomes in implantable cardioverter-defibrillator recipients: A systematic review. Cardiovasc Digit Health J. 3 , 46–55 (2022).

Rosman, L., Lampert, R., Sears, S. F. & Burg, M. M. Measuring physical activity with implanted cardiac devices: a systematic review. J Am Heart Assoc 7, e008663 (2018).

Kolk, M. et al. Behavioural digital biomarkers enable real-time monitoring of patient-reported outcomes: a substudy of the multicenter, prospective observational SafeHeart study. Eur Heart J Qual Care Clin Outcomes qcad069, (2023).

Hughes, A., Shandhi, M. M. H., Master, H., Dunn, J. & Brittain, E. Wearable devices in cardiovascular medicine. Circ. Res 132 , 652–670 (2023).

Kolk, M. Z. H. et al. Machine learning of electrophysiological signals for the prediction of ventricular arrhythmias: systematic review and examination of heterogeneity between studies. EBioMedicine 89 , 104462 (2023).

Article   PubMed   PubMed Central   Google Scholar  

Kolk, M. Z. H. et al. Dynamic prediction of malignant ventricular arrhythmias using neural networks in patients with an implantable cardioverter-defibrillator. EBioMedicine 99 , 104937 (2023).

Min, E. et al. A survey of clustering with deep learning: from the perspective of network architecture. IEEE Access 6 , 39501–39514 (2018).

Article   Google Scholar  

Karim, M. R. et al. Deep learning-based clustering approaches for bioinformatics. Brief. Bioinform 22 , 393–415 (2021).

Ginder, C. et al. Predicting malignant ventricular arrhythmias using real-time remote monitoring. J. Am. Coll. Cardiol. 81 , 949–961 (2023).

Shakibfar, S. et al. Predicting electrical storms by remote monitoring of implantable cardioverter-defibrillator patients using machine learning. Europace 21 , 268–274 (2019).

Soares-Miranda, L. et al. Physical activity and heart rate variability in older adults: the Cardiovascular Health Study. Circulation 129 , 2100–2110 (2014).

Elliott, A. D. et al. Association between physical activity and risk of incident arrhythmias in 402 406 individuals: evidence from the UK Biobank cohort. Eur. Heart J. 41 , 1479–1486 (2020).

Qiu, S. & Xing, Z. Association between accelerometer-derived physical activity and incident cardiac arrest. Europace 25 , euad353 (2023).

Yang, L. et al. Association of accelerometer-derived circadian abnormalities and genetic risk with incidence of atrial fibrillation. NPJ Digit Med 6 , 31 (2023).

Liebzeit, D., Phelan, C., Moon, C., Brown, R. & Bratzke, L. Rest-activity patterns in older adults with heart failure and healthy older adults. J. Aging Phys. Act. 25 , 116–122 (2017).

Castela Forte, J. et al. Identifying and characterizing high-risk clusters in a heterogeneous ICU population with deep embedded clustering. Sci. Rep. 11 , 12109 (2021).

Rowlands, A. V. et al. Beyond cut points: accelerometer metrics that capture the physical activity profile. Med Sci. Sports Exerc 50 , 1323–1332 (2018).

Frodi, D. M. et al. Rationale and design of the SafeHeart study: Development and testing of a mHealth tool for the prediction of arrhythmic events and implantable cardioverter-defibrillator therapy. Cardiovasc Digit Health J. 2 , S11–S20 (2021).

von Elm, E. et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J. Clin. Epidemiol. 61 , 344–349 (2008).

Esliger, D. W. et al. Validation of the GENEA accelerometer. Med Sci. Sports Exerc 43 , 1085–1093 (2011).

van Hees, V. T. et al. A novel, open access method to assess sleep duration using a wrist-worn accelerometer. PLoS One 10 , e0142533 (2015).

Spertus, J. A. & Jones, P. G. Development and Validation of a Short Version of the Kansas City Cardiomyopathy Questionnaire. Circ. Cardiovasc Qual. Outcomes 8 , 469–476 (2015).

Herdman, M. et al. Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual. Life Res 20 , 1727–1736 (2011).

Kingma, D. P. et al. Improved Variational Inference with Inverse Autoregressive Flow. Vol. 29 (eds Lee, D.et al.) (2016).

Higgins, I. et al. beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. in International Conference on Learning Representations (2016).

Vouros, A., Langdell, S., Croucher, M. & Vasilaki, E. An empirical comparison between stochastic and deterministic centroid initialisation for K-means variations. Mach. Learn. 110 , 1975–2003 (2021).

Hennig, C. Cluster-wise assessment of cluster stability. Computational Stat. Data Anal. 52 , 258–271 (2007).

Pedregosa, F. et al. Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12 , 2825–2830 (2011).

Google Scholar  

Scott, M. L. & Su-In, L. A Unified Approach to Interpreting Model Predictions. ArXiv abs/1705.07874 (2017).

Tianqi, C. & Carlos, G. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016).

Download references

Acknowledgements

This research was supported by the Horizon 2020 European Union funding programme for research and innovation (grant number: Eurostars project E!113994- SafeHeart). This research is partly funded by the Amsterdam Cardiovascular Sciences and research programme Rubicon which is financed by the Dutch Research Council (NWO).

Author information

Authors and affiliations.

Department of Clinical and Experimental Cardiology, Amsterdam UMC Location University of Amsterdam, Heart Center, Meibergdreef 9, Amsterdam, the Netherlands

Maarten Z. H. Kolk, Hanno L. Tan, Reinoud E. Knops & Fleur V. Y. Tjong

Amsterdam Cardiovascular Sciences, Heart Failure & Arrhythmias, Amsterdam UMC location AMC Meibergdreef 9, 1105 AZ, Amsterdam, the Netherlands

Maarten Z. H. Kolk, Reinoud E. Knops & Fleur V. Y. Tjong

Department of Cardiology, Copenhagen University Hospital Rigshospitalet, Inge Lehmanns Vej 7, 2100, Copenhagen, Denmark

Diana My Frodi, Peter Karl Jacobsen, Niels Risum, Jesper Hastrup Svendsen & Søren Zöga Diederichsen

Activinsights Ltd., Unit 11, Harvard Industrial Estate, Kimbolton, Huntingdon, PE28 0NJ, United Kingdom

Joss Langford

College of Life and Environmental Sciences, University of Exeter, Stocker Rd, Exeter, EX4 4PY, United Kingdom

Department of Computer Science, University of Copenhagen, Universitetsparken 1, 2100, Copenhagen, Denmark

Tariq O. Andersen

Netherlands Heart Institute, Moreelsepark 1, 3511 EP, Utrecht, The Netherlands

Hanno L. Tan

Department of Clinical Medicine, Faculty of Health and Medical Sciences, University of Copenhagen, Blegdamsvej 3B, 2200, Copenhagen, Denmark

Jesper Hastrup Svendsen

You can also search for this author in PubMed   Google Scholar

Contributions

M.K., R.K., J.H.S., J.L., T.A., S.D. and F.T. contributed to the conception and design of the study. M.K., D.F., J.L. and F.T. collectively investigated the data and decided on the methodology to be used. M.K., F.T. and J.L. conducted the formal analyses. M.K. and F.T. drafted the original manuscript. M.K., D.F., J.L., T.A., P.K.J., N.R., H.T., J.H.S., S.D., R.K. and F.T. reviewed, edited, and agreed with the final version of the manuscript. M.K., J.L., D.F. and F.T. accessed and verified the underlying data.

Corresponding author

Correspondence to Fleur V. Y. Tjong .

Ethics declarations

Competing interests.

The authors declare no competing non-financial interests but the following competing financial interests. R.E.K. reports consultancy fees and research grants from Boston Scientific, Medtronic and Abbott. and has stock options from AtaCor Medical Inc. F.V.Y.T. declares grants or contracts from the Dutch Research Council (NWO) and Amsterdam Cardiovascular Sciences, and received payment or honoraria from Boston Scientific and Abbott (paid to the institution). S.Z.D. reports consultancy fees and research grants from Acesion Pharma and Cortrium, and has received payment or honoraria from Bristol, Myers Squibb, Pfizer and Bayer. P.K.J. reports consultancy fees and research grants from Abbott and Medtronic, payment or honoraria from Abbott and Medtronic and support for attending meetings and/or travel from Abbott and Medtronic. J.H.S. reports grants or contracts from Medtronic (payed to institution), payment or honoraria from Medtronic, support for attending meetings and/or travel from Abbott and Medtronic, participation on Medtronic Advisory Board, and stocks or stock options from Vital Beats. J.L. reports having stock or stock option from Activinsights Ltd. T.O.A. reports having stock or stock options from Vital Beats. D.M.F. reports financial support for attending meetings and/or travel from Boston Scientific. The authors M.Z.H.K., N.R. and H.L.T. declare no competing financial or non-financial interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary material, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Kolk, M.Z.H., Frodi, D.M., Langford, J. et al. Deep behavioural representation learning reveals risk profiles for malignant ventricular arrhythmias. npj Digit. Med. 7 , 250 (2024). https://doi.org/10.1038/s41746-024-01247-w

Download citation

Received : 12 January 2024

Accepted : 30 August 2024

Published : 16 September 2024

DOI : https://doi.org/10.1038/s41746-024-01247-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.