distance education research paper

• Research article
• Open access
• Published: 20 May 2020

Students’ perceptions on distance education: A multinational study

• Patricia Fidalgo 1 ,
• Joan Thormann 2 ,
• Oleksandr Kulyk 3 &
• José Alberto Lencastre 4  

International Journal of Educational Technology in Higher Education volume  17 , Article number:  18 ( 2020 ) Cite this article

363k Accesses

128 Citations

12 Altmetric

Metrics details

Many universities offer Distance Education (DE) courses and programs to address the diverse educational needs of students and to stay current with advancing technology. Some Institutions of Higher Education (IHE) that do not offer DE find it difficult to navigate through the steps that are needed to provide such courses and programs. Investigating learners’ perceptions, attitudes and willingness to try DE can provide guidance and recommendations for IHEs that are considering expanding use of DE formats. A survey was distributed to undergraduate students in Portugal, UAE and Ukraine. The results of this pilot study showed that in all three countries, students’ major concerns about such programs were time management, motivation, and English language skills. Although students were somewhat apprehensive many indicated they were interested in taking DE courses. Six recommendations informed by interpretation of students’ responses and the literature, are offered to assist institutions who want to offer DE as part of their educational strategy.

Introduction

The World Wide Web has made information access and distribution of educational content available to a large fraction of the world’s population and helped to move Distance Education (DE) to the digital era. DE has become increasingly common in many universities worldwide (Allen & Seaman, 2017 ). Nonetheless, there are still many universities that do not provide this opportunity because it is not part of their institutional culture. As DE becomes more prevalent, countries and Institutions of Higher Education (IHE) that do not provide DE courses will need to look at this option to retain and expand their student population. (Keegan, 1994 ; Nakamura, 2017 ).

In order to develop such programs, it is useful to determine if students are receptive to taking such online courses and are prepared to do so. This study addresses students’ perceptions and their interest in DE. In addition, it provides a comparative analysis across three countries whose IHEs do not have extensive offerings in DE. The results of this research provide some strategies to encourage and support students to take DE courses.

Literature review

A seminal article by Keegan ( 1980 ) presents key aspects of DE. Some of the elements are: physical separation of teacher and learner, learning occurs in the context of an educational institution, technical media are used, teacher and learner communicate, face to face meetings are possible, and an industrial model of providing education is used. More recently varying definitions of DE seem to be based on the perspective of various educators and to reflect the educational culture of each country and IHE. However, some common descriptors seem to be accepted by most stakeholders in the field. Distance education is an educational experience where instructors and learners are separated in time and space (Keegan, 2002 ) which means it can happen away from an academic institution and can lead to a degree or credential (Gunawardena, McIsaac, & Jonassen, 2008 ).

Although there are different types of DE, this research focuses on online learning. The following types of online learning will be investigated: synchronous, asynchronous, blended, massive online open courses (MOOC), and open schedule online courses. In synchronous instruction, teachers and learners meet (usually online) for a session at a predetermined time. According to Watts ( 2016 ) live streaming video and/or audio are used for synchronous interaction. Although videoconferencing allows participants to see each other this is not considered a face-to-face interaction because of the physical separation (Keegan, 1980 ).

Asynchronous instruction means that teachers and learners do not have synchronous sessions and that students have access to course content through the Internet at any time they want or need. Communication among the participants occurs mainly through email and online forums and is typically moderated by the instructor (Watts, 2016 ). According to Garrison ( 2000 ) “Asynchronous collaborative learning may well be the defining technology of the postindustrial era of distance education.” (p.12) Yet another type of DE is blended learning (BL). Garrison and Kanuka ( 2004 ) define BL as combining face-to-face classroom time with online learning experiences. Although it is not clear as to how much time is allocated to online in the blended model “the real test of blended learning is the effective integration of the two main components (face-to-face and Internet technology) such that we are not just adding on to the existing dominant approach or method.” (p.97) In the BL format different teaching strategies and instructional technology can be used to help individuals who have different learning styles, needs and interests (Tseng & Walsh Jr., 2016 ).

Another type of DE is MOOCs (Massive Online Open Courses). This format was first introduced in 2006 and offers distributed open online courses that are available without cost to a very large number of participants (Cormier, McAuley, Siemens, & Stewart, 2010 ). MOOCs origins can be traced to the Open Access Initiative in 2002 which advocates sharing knowledge freely through the Internet. By providing educational opportunities MOOCs can address the increasing demand for training and education (Zawacki-Richter & Naidu, 2016 ). Finally, in open schedule online courses students work asynchronously with all the materials being provided digitally. Although there are deadlines for submitting assignments, students working at their own pace have some independence as to when they do their coursework (Campus Explorer, 2019 ).

There are advantages and disadvantages in taking DE courses. Some of the advantages are self-paced study, time and space flexibility, time saving (no commute between home and school) and the fact that a distance learning course often costs less. Disadvantages include a sense of isolation, the struggle with staying motivated, lack of face-to-face interaction, difficulty in getting immediate feedback, the need for constant and reliable access to technology, and occasionally some difficulty with accreditation (De Paepe, Zhu, & Depryck, 2018 ; Lei & Gupta, 2010 ; Venter, 2003 ; Zuhairi, Wahyono, & Suratinah, 2006 ).

Most of the literature concerning student perception of DE courses, both blended and entirely online, involves students who have enrolled in online courses. Some articles address comparisons of perceptions between face-to-face and online students regarding DE (Daniels & Feather, 2002 ; Dobbs, del Carmen, & Waid-Lindberg, 2017 ; Hannay & Newvine, 2006 ; Lanier, 2006 ). Additional studies address adult and undergraduate students and cover many aspects of the online experience (Dobbs et al., 2017 ; Horspool & Lange, 2012 ; Seok, DaCosta, Kinsell, & Tung, 2010b , a ). However, little, if any research has been conducted that only addresses perceptions of students who live in countries in which few IHEs offer online courses.

In a study comparing online and face-to-face learning, Horspool and Lange ( 2012 ) found that students chose to take online courses to avoid travel time to class and scheduling problems. A majority of both face-to-face and online students did not experience technological issues. Both groups also found that communication with the instructor was adequate. Online students indicated that instructor response time to questions was prompt. By contrast online students perceived peer communication as occurring much less often. Course satisfaction was comparable for both formats (Horspool & Lange, 2012 ). Responses to another survey concerning online and traditional course formats found that students’ reasons for taking online courses included flexibility to accommodate work and family schedules, the ability to avoid commuting to the university and more online courses being available to them (Dobbs et al., 2017 ). Both online and traditional students agreed that traditional courses were easier, and they learned more in that format. They also concurred that online courses required more effort. Experienced online students indicated that the quality of their courses was good while traditional students who had never taken an online course felt that the quality of online courses was lower.

There is additional research that focuses on students including those enrolled in community colleges, MOOCs, blended learning as well as adult learners. Community college students’ and instructors’ perceptions of effectiveness of online courses were compared by Seok et al. ( 2010b , a ). The researchers focused on pedagogical characteristics (management, Universal Design for Learning, interaction, teaching design and content) and technical features (interface, navigation and support). In addition, responses were examined based on various aspects of the subjects’ demographics. Two surveys with 99 items were distributed electronically. One survey was for instructors and the other for students. In general, instructors and students indicated that teaching and learning online was effective. Female students responded more positively to most questions concerning effectiveness, and instructors also found it more positive (Seok et al., 2010b , a ).

Students who enrolled in a MOOC were motivated to take other courses in this format based on their perception that it was useful for achieving their goals. In addition, their motivation was high if the course was posted on a platform that was easy to use (Aharony & Bar-Ilan, 2016 ). This study also found that as students proceeded through the course, they gained confidence.

Blended learning was examined by Kurt and Yildirim ( 2018 ) to determine student satisfaction and what they considered to be important features of the blended format. The results indicated that the Turkish students who participated, almost unanimously felt that BL was beneficial and that their own role and the instructors’ role was central to their satisfaction. The authors stated, “the prominent components in the process have been identified as face-to-face lessons, the features of online course materials, LMS used, design-specific activities, process-based measurement and evaluation, student-student interaction and out-of-class sharing respectively.” (p. 439) DE has a growth potential and offers the opportunity to reach many people (Fidalgo, 2012 ), hence it can be used as a technique for mass education (Perraton, 2008 ). According to Perraton ( 2008 ) DE can be adapted to the needs of current and previous generations who did not complete their education. DE can also reach individuals who live in remote locations and do not have the means to attend school.

Methodology

Study goals.

The goal of this pilot study is to examine what undergraduate students’ perceptions are concerning DE and their willingness to enroll in this type of course. This study focuses on three countries that do not offer extensive DE accredited programs. By comparing three countries with similar DE profiles, commonalties and differences that are relevant and useful can be found. When the IHEs from these countries decide or have the conditions to move towards DE, the results of this study may help them adapt this format to their particular context and students’ needs. Results may also help IHEs plan their strategy for offering online courses to current and future students and attract prospective students who otherwise would not be able to enroll in the face-to-face courses that are available.

Research questions

Have undergraduate students taken an online course previously?

What are undergraduate students’ perceptions of distance education?

What are the reasons for undergraduate students to enroll/not enroll is distance education courses?

What preparation do undergraduate students feel they need to have before taking distance education courses?

What is the undergraduate students’ receptivity towards enrolling in distance education courses?

What types of distance education would undergraduate students be interested in taking?

This research was conducted at IHEs in three countries (Portugal, Ukraine and UAE). A description of each country’s sociodemographic and technological use provides a context for this study.

Portugal, a country located at the western end of the European continent, has a resident population of just over 10 million people (Instituto Nacional de Estatistica, 2019 ). Data collected by Instituto Nacional de Estatistica in 2019 indicated that almost 81% of households in Portugal had Internet access at home. According to the Portuguese National Statistical Institute ( 2019 ), the rate of Internet use by the adult population is about 76%. Among this population, people who attend or have completed secondary and higher education have a higher percentage of Internet use (98%) (Instituto Nacional de Estatistica, 2019 ).

The most used devices to access the Internet are smartphones and laptops. Regarding computer tasks, the most frequent ones are copying and moving files and folders and transferring files from the computer to other devices (PORDATA - Base de Dados Portugal Contemporâneo, 2017 ).

Among Internet users, 80% use social networks, which is a higher percentage than the European Union (EU) average. Mobile Internet access (outside the home and workplace and on portable devices) is 84% and maintains a strong growth trend (Instituto Nacional de Estatistica, 2019 ).

Ukraine is one of the post-soviet countries located in Eastern Europe and it strives to be integrated in economic and political structures of the EU. The current population of the country is 42 million. Despite the low incomes of many Ukrainians, modern technological devices are widespread among the population. The State Statistics Service of Ukraine ( 2019 ) reported that there were 26 million Internet subscribers in the country in the beginning of 2019. However, Ukrainians do not have a high level of digital literacy yet. According to the Digital Transformation Ministry of Ukraine (Communications Department of the Secretariat of the CMU, 2019 ), almost 38% of Ukrainian people aged from 18 to 70 have poor skills in computer literacy and 15.1% of the citizens have no computer skills.

According to the survey conducted by the Digital Transformation Ministry of Ukraine (The Cabinet of Ministers of Ukraine, 2019 ) 27.5% Ukrainian families have a tablet, and 30.6% have one smart phone, 26.4% have two smart phones, 16.5% have three smart phones and 10.8% have four and more smart phones. As for laptops, 42.7% Ukrainian families have a laptop and 45.6% have a desktop computer (The Cabinet of Ministers of Ukraine, 2019 ). The data from the ministry did not indicate if families have multiple devices, however the data shows that technological devices are widespread.

The United Arab Emirates (UAE) is a country located in the Persian Gulf that borders with Oman and Saudi Arabia. The UAE has a population of 9.77 million and is one of the richest countries in the world based on gross domestic product (GDP) per capita. The resident population consists of 11,5% Emiratis and the remaining residents are expats from countries such as India, Pakistan, Philippines, Egypt and others (Global Media Insight, 2020 ).

Regarding technology use, 91% of the residents use mobile Internetand over 98% of the households have Internet access (Knoema, 2018 ). Mobile devices such as smartphones are used to access the Internet mainly at home or at work (Federal Competitiveness and Statistics Authority, 2017 ).

In 2017 the most frequent Internet activities were: sending/receiving emails (61%), posting information or instant messaging (55%), getting information about goods or services (45%), reading or downloading online newspapers, magazines or electronic books (41%) and telephoning over the Internet/VOIP (33%). Downloading movies, images, music, watching TV or video, or listening to radio or music is also a frequent activity performed by 27% of the Internet users followed by Internet banking (25%) and purchasing or ordering good and services (22%) (Federal Competitiveness and Statistics Authority, 2017 ).

While these three countries were selected due to the location of the researchers and thus provided convenience samples, the three countries have a similar lack of DE offerings. Online surveys were emailed to students enrolled in a variety of undergraduate face-to-face courses during the fall semester of 2018. The students in Portugal and the UAE were enrolled in a teacher education program and the survey was emailed to two course sections in Portugal (73 students) and four course sections in the UAE (108 students). At the IHE in Ukraine, students were majoring in applied mathematics, philology, diagnostics, social work and philosophy, and surveys were emailed to 102 students who were enrolled in five course sections. In Portugal and Ukraine, the URL for the online survey was emailed by the instructor of all the course sections. In the UAE the instructor who emailed the URL for the survey taught two of the course sections. The students in the other two sections knew this instructor from taking courses with her previously. The students participating in this study were a convenience sample based on the disciplines taught by the researchers.

Data collection

An online survey with 10 closed questions about undergraduate students’ perception and receptivity towards enrolling in DE courses was developed by the researchers. Ary, Jacobs, Sorensen, and Walker ( 2010 ) compared traditional methods (i.e. face-to-face, paper and pencil) with web-based surveys and found the latter to be are more effective for gathering data from many participants. The questions designed by the researchers were informed by their experience/practice as well as in-depth literature review. The survey was created to respond to the research questions that guided this study. Response choices to the multiple-choice questions were based on issues and concerns related to DE. Students’ responses were collected towards the end of the first semester of the 2018/19 academic year.

The survey was developed to address research questions that assess undergraduate students’ perceptions of DE and students’ receptivity towards enrolling in DE courses (c.f. Appendix ). The use of surveys allows researchers to “obtain information about thoughts, feelings, attitudes, beliefs, values, perceptions, personality and behavioral intentions of research participants.” (Johnson & Christensen, 2014 , p. 192) The survey questions included multiple response formats: Likert scale, select more than one response and multiple choice. Surveys for Portugal were presented in Portuguese. In Ukraine the surveys were translated into Ukrainian. Since English is the language of instruction at the UAE institution, their survey was in English. The URL for the survey was emailed to students by their instructors and was available in an online Google Form. The survey took approximately 10 min to complete. The study consisted of a “self-selected” convenience sample.

Data analysis

Survey results were recorded in Google Forms and an Excel spreadsheet was used to collect students’ responses. Descriptive statistics of the responses to the survey are presented in graphs and tables with percentages of responses displayed. The descriptive statistics provide summaries about the sample’s answers to each of the questions as well as measures of variability (or spread) and central tendency.

Research approval and data management

The research proposal was submitted to the Research and Grants Committee and approved by the Institutional Review Board of the college in the UAE. No personal information (name, College ID number or any other type of information that allows the identification of students) was asked from the students in the surveys. The surveys were anonymous. Only the Principal Investigator (PI) had access to all the data collected. The data will be stored in the PI’s password protected computer for 5 years.

Fifty five of the 73 Portuguese students who received the survey responded and 98 of the 108 UAE students responded. In the Ukraine 102 students were sent surveys and 70 responded. Below are participants’ responses to questions concerning age, gender, as well as level of confidence using the computer and the Internet.

Students’ age range was from 17 to 50 years old. Most students’ age ranges were between 17 and 29 years. Survey responses indicated that 7% of the students in the UAE were male and 93% female, in the Ukraine 43% were male and 57% female and in Portugal 9% male, and 91% female.

Participants were asked about their level of confidence using a computer and the Internet. Results are presented in Table  1 .

The use of participants from three countries allows the study of trends and to determine differences and/or similarities of perceptions about DE. Although the students were enrolled in courses in diverse content areas, they were all undergraduates, almost all under 30 years old, and most were confident using the computer and Internet. These demographic similarities provided a relatively cohesive group for this study while allowing a comparison across countries.

A range of questions were asked about students’ attitudes towards and experience with DE. To determine the participants’ experience with DE two questions were asked.

The data indicates that out of 223 students who responded to the survey, a total of 63 students have taken DE courses. Half of the Ukraine students, about one quarter of the UAE students and only 5% of students in the group from Portugal had taken DE courses (Fig.  1 ). As shown in Fig.  2 , of the students who have had previous experience in DE, most Ukraine students have taken one or two online courses, most UAE students have taken one course and a few Portuguese students have taken one course.

Students that have taken distance education courses

Number of distance education courses taken

More than half of Portuguese students, about two thirds of the Ukraine students and a little over one third of UAE students had a Very favorable or Favorable attitude towards DE. Approximately one third of Portuguese and Ukraine students were Neutral/Unable to judge their attitude. A little less than half of UAE students also indicated this. A small percentage of Portuguese, and one fifth of UAE students indicated their attitude was Very unfavorable or Unfavorable and no Ukraine students reported this (Table 2 ).

More than one third of Portuguese students shared that managing class and study time, saving time by choosing study location and working at their own pace were reasons to enroll in DE. About two thirds of the students from Ukraine reported that working at their own pace and managing their study time were reasons to enroll. A little more than half of these students reported that reasons for enrolling in DE included managing class time, saving time by selecting study location and not having to travel to school as well as having more options for courses or colleges to attend. Almost half of the UAE students had similar reasons for enrolling in a DE courses including managing class and study time, saving time by choosing study location and working at their own pace. In addition, a little more than half of the UAE students also shared that having more options for courses or colleges to attend were reasons to enroll. The reasons that were selected the least by all three groups were that courses were less expensive and enrolling in a preferred program (Tables  3 and 4 ).

Students were given eleven options as to why they would not enroll in DE courses, which are displayed in Tables  5 and 6 . Two reasons that were chosen most often were difficulty staying motivated and preferring face-to-face classes. A small number of Ukraine students reported this as a reason to not enroll in DE courses. Difficulty getting immediate feedback was also a concern for UAE students. Close to one third in the three groups indicated that difficulty contacting the instructor and interacting with peers as well as missing campus life are reasons for not enrolling. About one tenth of Portuguese, one fifth of Ukraine and one fifth of the UAE students reported difficulty getting accreditation as a reason for not enrolling. Not knowing enough about DE was indicated by one tenth of Portuguese, one fifth of Ukraine and one fifth of the UAE students. Only a small number of all the students indicated three categories that are frequently cited in the literature as preventing students from enrolling, these include access to technology, feeling of isolation and too great an expense.

Tables  7 and 8 show student responses to a question regarding the preparation they think they would need before enrolling in a DE course. A little over one tenth of the Portuguese students indicated that they needed better computer equipment, writing skills and a dedicated study space. About one quarter of these students reported they need better skills in the following areas: time management, computer and English language skills, as well as needing to have learning goals and objectives. Having a better Internet connection and the need to develop a study plan was shared by approximately one third of these students. Finally, the highest rated prerequisite for these Portuguese students was to be more motivated.

Few of the Ukraine students felt that they needed better computer equipment or skills, a dedicated study space or a better Internet connection at home. Their concerns focused on their behaviors as students since half or a little more than half felt they needed to be more motivated, have learning objectives and goals, a study plan and better management skills. About one third of these students also reported that they needed better English language skills.

The UAE students were less confident than the Ukraine students about computer skills and needing better equipment and a better Internet connection at home. Almost half of these UAE students reported their need for a study plan and motivation as their most pressing needs. Better management and English language skills were recorded by about one third of the students. One quarter of the UAE students felt they needed better writing skills and a dedicated study space.

Table 9 shows students’ interest in enrolling in DE courses. Almost one quarter of the Ukraine students are Extremely interested in taking DE courses and almost half are Somewhat interested. This contrasts with the students from Portugal who indicated that only 5% are Extremely interested and almost a quarter Somewhat interested. The UAE students’ interest in enrolling fell in between the students from the two other countries. One fifth to almost one third of all three groups were Neutral/Unable to judge. About one tenth of students from Ukraine reported Not being very interested or Not at all interested which contrasts with the Portuguese and UAE students whose numbers were about one half and one quarter respectively.

Tables  10 and 11 show the types of DE that the students were interested in trying. Portuguese students favored Open schedule courses, followed by Blended learning and Synchronous. Few of these students were interested in MOOCs and Asynchronous. More than half of the students from Ukraine were interested in MOOCs and Blended learning followed by Open schedule. About one third of these students were interested in Synchronous and Asynchronous. UAE students most popular formats were Open schedule and Blended learning followed by Synchronous and Asynchronous. There was little interest in MOOCs by the UAE students. Few Portuguese and Ukraine students indicated that they would not take a DE course, however, almost a quarter of the UAE students indicated this.

Data indicates close to a 100% of the UAE residents use the Internet at home or on their mobile devices (Knoema, 2018 ). By contrast a smaller percentage of individuals use the Internet in Portugal and the Ukraine (Infographics, 2019 ). Internet use in each country does not seem to greatly impact UAE students’ opinions regarding DE.

Students’ perceptions of DE vary across the participants from the three countries. Portuguese and Ukrainian students rated DE more favorably than UAE students. Half of the Ukrainian students have experience with DE which might account for their favorable attitude. In contrast, in Portugal only a very small percentage of the students had experience. However, this does not seem to have negatively influenced their attitude towards DE. The interest level and engagement with new technologies by Portuguese students may help explain the favorable perception the participants had toward DE. A study by Costa, Faria, and Neto ( 2018 ) found that 90% of Portuguese students use new technologies and 69% of them use new technologies more than an hour and a half a day. Based on three European studies, Diário de Noticias ( 2011 ) stated that Portuguese students “appear at the forefront of those who best master information and communication technologies (ICT).” (para.1) Another factor influencing respondents might be that currently, and for the first time, the Portuguese government has passed a law that will regulate DE in the country. This new law will open the possibility for other IHEs to provide DE courses that lead to a degree.

Ukrainian students reported a high level of confidence in operating technological devices. The reason for this may be, in part, because of state educational requirements. Since the end of the 1990s, all Ukrainian students in secondary schools have at least one computer course as a mandatory element of their curriculum. This course covers a wide range of issues, which vary from information society theory to applied aspects of computer usage. Among the seven learning goals of this course three address digital literacy (Ministry of Education and Science of Ukraine, 2017 ). Ukrainian students who responded to the survey have taken computer courses for at least 5 years.

In the UAE, most DE courses and programs are not accredited by the Ministry of Education (United Arab Emirates Ministry of Education, 2016 ), which may account for UAE students lack of experience and their inability to judge this type of instruction.

It is worth analyzing the reasons why students enrolled or would enroll in DE courses. The reasons for taking DE courses, such as time management issues, are supported by studies concerning self-regulation and higher retention rates (Bradley, Browne, & Kelley, 2017 ; Peck, Stefaniak, & Shah, 2018 ). Students’ interest in having more control of their study time is also mentioned as one of the primary benefits of DE (Alahmari, 2017 ; Lei & Gupta, 2010 ). Regarding the reasons for not enrolling in DE courses, participants from the three countries mentioned difficulty contacting instructors and peers. Also, more than half of the students in Portugal and the UAE indicated they preferred face-to-face classes. Most students have spent their entire academic lives in traditional classes where interaction and immediate feedback from instructors and peers are more common. These concerns may be why students perceive they would lose a familiar type of interaction and have to engage with classroom participants in a new and different way (Carver & Kosloski Jr., 2015 ; Morris & Clark, 2018 ; Robinson & Hullinger, 2008 ; Summers, Waigandt, & Whittaker, 2005 ). It should be noted that the Portuguese and UAE students were enrolled in teacher education programs and are training to be face-to-face teachers. They may not understand the potential of DE format and are not preparing or expecting to use DE in their professional careers.

Difficulty being motivated was another reason chosen by the participants of the three countries to not enroll in DE courses. The lack of experience in this type of educational format may help explain student lack of confidence with their ability to study and stay on task. This response contrasts with the reasons reported for enrolling in DE courses such as controlling their study time. On one hand, participants like the prospect of having the ability to manage their own time. On the other hand, they are concerned they may lack the discipline they need to be successful.

Although the literature indicates that access to technology, isolation and expense are reasons frequently cited as preventing students from enrolling in DE courses (Lei & Gupta, 2010 ; Venter, 2003 ; Zuhairi et al., 2006 ), these reasons were selected by a very small percentage of the participants of this study. Access and affordability of technology has rapidly increased over the last decade which may help explain this inconsistency. Students may understand that DE courses are now less expensive than traditional university courses (Piletic, 2018 ) and they do not cite this as a reason for not enrolling. Relatively few students indicated they would feel isolated. Since this generation is in constant communication using technology (Diário de Notícias, 2011 ) they may not associate DE learning with isolation. However, it is interesting to note that there was a greater concern for interacting with instructors and peers than isolation.

The Ukrainian students are the most receptive to enrolling in DE courses. This is consistent with their positive perception of this type of learning. In addition, the previous experience of half of the participants may influence their interest as well as encourage their peers’ receptivity. UAE students do not have much experience and fewer than half are open to enrolling in DE courses. This may be due to their lack of experience and other concerns previously mentioned. Only one third of the Portuguese participants indicated their interest in enrolling in DE courses. This is in contrast with almost two thirds saying they had a favorable or very favorable attitude. The reasons for this inconsistency are not evident.

In terms of preparation needed to take DE courses, technical concerns were less of an issue for the participants of all three countries than skills and behaviors. Most participants’ answers focused on student skills including computer, English language and time management. Behaviors such as developing a study plan, having learning goals and objectives and being more motivated were also mentioned. The perceived need for better English language skills was expressed by about one third of the participants, none of whom have English as their native language. English speaking countries have been dominant in DE making English the most commonly used language in online learning (Sadykova & Dautermann, 2009 ). Regarding time management, half of the Ukrainian students expressed their need for improvement in contrast to approximately one third of the participants from the other countries. The difference among responses may be because the Ukrainian students are more self-reflective, or the others are more disciplined. Although both DE and face-to-face courses have deadlines for tasks and assessments, in the face-to-face courses, students meet in person with their instructors who may support and press them to do their work. Lack of in person contact may account for the participants feeling they need to improve these skills when taking DE courses (De Paepe et al., 2018 ). Students expressed concerns about lacking certain skills and having certain behaviors that would lead them to be reluctant to enroll in DE courses. The need for help and preparation are some of the concerns that participants reported. Perceived needs may account for the students’ apprehensions regarding taking DE courses. To promote this type of instruction, IHEs could address students’ concerns (Mahlangu, 2018 ).

Open schedule and blended learning courses were the two preferred formats stated by the participants. The reason that Open schedule is the most popular may be that it provides more freedom than other types of courses. Blended learning offers the familiar face-to-face instruction and some of the conveniences of DE which may be why participants are interested in this model.

Studies regarding the use of MOOCs in all three countries have been conducted indicating that researchers in these locations are aware that this course format is of potential interest to local students (Eppard & Reddy, 2017 ; Gallacher, 2014 ; Gonçalves, Chumbo, Torres, & Gonçalves, 2016 ; Sharov, Liapunova, & Sharova, 2019 ; Strutynska & Umryk, 2016 ). Ukrainian students selected MOOCs much more than students in the other countries. The reason for this may be that these students are more knowledgeable about MOOCs, because this type of course is usually at no cost and/or offered by prestigious IHEs (Cormier et al., 2010 ). However, this study did not ask why students were interested in MOOCs or other types of DE courses.

Limitations and future research

While this study offers useful information regarding undergraduate students’ perception and receptivity in taking DE courses, it has limited generalizability because of the size of the sample and the type of statistical analysis performed. Participants from two of the countries were enrolled in teacher education programs and were primarily female, thus future studies would benefit from including more students in diverse programs and a more equitable gender distribution.

Since many IHEs also offer programs for graduate students it would be useful to survey these students about their opinion and availability to enroll in DE courses. This would provide additional information for IHEs that are interested in developing DE programs.

There were some inconsistencies in the students’ responses such as Portuguese students’ interest in enrolling in DE courses not matching their favorable/ very favorable attitude towards DE. It would be helpful to conduct future research regarding this and other inconsistencies.

A study is currently being planned to collect data that will provide a larger and more diverse sample and include additional IHEs. This future research will potentially increase the available knowledge on how to provide DE for a greater number of students.

Conclusion and recommendations

Further development of DE courses and programs at IHEs in countries such as Portugal, UAE and Ukraine have good prospects. The students’ primary concerns regarding taking DE courses were similar among the three countries. These concerns included time management, motivation, and English language skills. However, this did not totally diminish participants interest in taking online courses especially for the Ukrainian students.

Based on this research, there are some obstacles that can be addressed to support the expansion of DE in the three countries that were studied and in other countries. The following recommendations may assist IHEs in promoting DE.

Recommendations for preparation within IHEs

IHEs can take proactive steps to prepare DE offerings, however, a one-size fit all model may not be appropriate for all countries and IHEs. Each institution needs to develop their own plan that meets the needs of their students and faculty. Data from this pilot study and the literature (Elbaum, McIntyre, & Smith, 2002 ; Hashim & Tasir, 2014 ; Hux et al., 2018 ) suggest that following steps might be taken:

Assess readiness to take DE courses through a survey and have students speak with counselors.

Provide pre-DE courses to build skills and behaviors based on students’ concerns.

Train instructors to develop and deliver DE courses that help to overcome obstacles such as motivation and time management.

Offer courses in a blended learning format to familiarize students with online learning which may provide a transitional model.

Recommendations for IHE outreach

This study shows that there is some student interest in enrolling in online courses. It is not sufficient for IHEs to make changes internally within their own institution. IHEs need to develop external strategies and actions that help advance the development of DE:

Promote DE in social media to target potential students and encourage them to take courses.

Urge government agencies to accredit DE courses and programs.

This pilot study provides some background information that may help IHEs to offer DE courses. Additional research about students’ preferences and needs regarding DE should be conducted. The sample size, IHEs included and participating countries could be expanded in order to gain a greater understanding.

Different cultural characteristics need to be taken into account in the development of online courses and programs. DE is being increasingly included by IHEs all around the world. To stay current, universities will need to find ways to offer DE to their current and prospective students.

Acknowledgements

Not applicable.

This research was not funded.

Author information

Authors and affiliations.

Curriculum and Instruction Division, Emirates College for Advanced Education, Abu Dhabi, United Arab Emirates

Patricia Fidalgo

Educational Technology Division, Lesley University, Cambridge, MA, USA

Joan Thormann

Philosophy Department, Oles Honchar Dnipro National University, Dnipropetrovs’ka oblast, Ukraine

Oleksandr Kulyk

Department of Curricular Studies and Educational Technology, University of Minho, Braga, Portugal

José Alberto Lencastre

You can also search for this author in PubMed   Google Scholar

Contributions

Patricia Fidalgo: design of the work, data collection, analysis, interpretation of data, and draft of the work. Joan Thormann: design of the work, analysis, interpretation of data, and draft of the work. Oleksandr Kulyk: data collection, interpretation of data, and draft of the work. José Alberto Lencastre: data collection. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Patricia Fidalgo .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Online Survey Questions

1. If the students have taken any distance education courses previously and if yes, how many;

2. What are the students’ perceptions of distance education;

3. What are the reasons students would enroll in distance education courses;

4. What are the reasons students would not enroll in a distance education course;

5. What preparation do students feel they need before taking distance education courses;

6. What is the level of students’ interest towards enrolling in distance education courses;

7. What types of distance education would students be interested in trying;

8. What is the students’ age;

9. What is the students’ gender;

10. How confident do students feel using a computer and the Internet.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Fidalgo, P., Thormann, J., Kulyk, O. et al. Students’ perceptions on distance education: A multinational study. Int J Educ Technol High Educ 17 , 18 (2020). https://doi.org/10.1186/s41239-020-00194-2

Download citation

Received : 11 December 2019

Accepted : 18 March 2020

Published : 20 May 2020

DOI : https://doi.org/10.1186/s41239-020-00194-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Distance education
• Multinational study
• Perceptions of distance education
• Undergraduate students

• Open supplemental data
• Reference Manager
• Simple TEXT file

People also looked at

Original research article, insights into students’ experiences and perceptions of remote learning methods: from the covid-19 pandemic to best practice for the future.

• 1 Minerva Schools at Keck Graduate Institute, San Francisco, CA, United States
• 2 Ronin Institute for Independent Scholarship, Montclair, NJ, United States
• 3 Department of Physics, University of Toronto, Toronto, ON, Canada

This spring, students across the globe transitioned from in-person classes to remote learning as a result of the COVID-19 pandemic. This unprecedented change to undergraduate education saw institutions adopting multiple online teaching modalities and instructional platforms. We sought to understand students’ experiences with and perspectives on those methods of remote instruction in order to inform pedagogical decisions during the current pandemic and in future development of online courses and virtual learning experiences. Our survey gathered quantitative and qualitative data regarding students’ experiences with synchronous and asynchronous methods of remote learning and specific pedagogical techniques associated with each. A total of 4,789 undergraduate participants representing institutions across 95 countries were recruited via Instagram. We find that most students prefer synchronous online classes, and students whose primary mode of remote instruction has been synchronous report being more engaged and motivated. Our qualitative data show that students miss the social aspects of learning on campus, and it is possible that synchronous learning helps to mitigate some feelings of isolation. Students whose synchronous classes include active-learning techniques (which are inherently more social) report significantly higher levels of engagement, motivation, enjoyment, and satisfaction with instruction. Respondents’ recommendations for changes emphasize increased engagement, interaction, and student participation. We conclude that active-learning methods, which are known to increase motivation, engagement, and learning in traditional classrooms, also have a positive impact in the remote-learning environment. Integrating these elements into online courses will improve the student experience.

Introduction

The COVID-19 pandemic has dramatically changed the demographics of online students. Previously, almost all students engaged in online learning elected the online format, starting with individual online courses in the mid-1990s through today’s robust online degree and certificate programs. These students prioritize convenience, flexibility and ability to work while studying and are older than traditional college age students ( Harris and Martin, 2012 ; Levitz, 2016 ). These students also find asynchronous elements of a course are more useful than synchronous elements ( Gillingham and Molinari, 2012 ). In contrast, students who chose to take courses in-person prioritize face-to-face instruction and connection with others and skew considerably younger ( Harris and Martin, 2012 ). This leaves open the question of whether students who prefer to learn in-person but are forced to learn remotely will prefer synchronous or asynchronous methods. One study of student preferences following a switch to remote learning during the COVID-19 pandemic indicates that students enjoy synchronous over asynchronous course elements and find them more effective ( Gillis and Krull, 2020 ). Now that millions of traditional in-person courses have transitioned online, our survey expands the data on student preferences and explores if those preferences align with pedagogical best practices.

An extensive body of research has explored what instructional methods improve student learning outcomes (Fink. 2013). Considerable evidence indicates that active-learning or student-centered approaches result in better learning outcomes than passive-learning or instructor-centered approaches, both in-person and online ( Freeman et al., 2014 ; Chen et al., 2018 ; Davis et al., 2018 ). Active-learning approaches include student activities or discussion in class, whereas passive-learning approaches emphasize extensive exposition by the instructor ( Freeman et al., 2014 ). Constructivist learning theories argue that students must be active participants in creating their own learning, and that listening to expert explanations is seldom sufficient to trigger the neurological changes necessary for learning ( Bostock, 1998 ; Zull, 2002 ). Some studies conclude that, while students learn more via active learning, they may report greater perceptions of their learning and greater enjoyment when passive approaches are used ( Deslauriers et al., 2019 ). We examine student perceptions of remote learning experiences in light of these previous findings.

In this study, we administered a survey focused on student perceptions of remote learning in late May 2020 through the social media account of @unjadedjade to a global population of English speaking undergraduate students representing institutions across 95 countries. We aim to explore how students were being taught, the relationship between pedagogical methods and student perceptions of their experience, and the reasons behind those perceptions. Here we present an initial analysis of the results and share our data set for further inquiry. We find that positive student perceptions correlate with synchronous courses that employ a variety of interactive pedagogical techniques, and that students overwhelmingly suggest behavioral and pedagogical changes that increase social engagement and interaction. We argue that these results support the importance of active learning in an online environment.

Materials and Methods

Participant pool.

Students were recruited through the Instagram account @unjadedjade. This social media platform, run by influencer Jade Bowler, focuses on education, effective study tips, ethical lifestyle, and promotes a positive mindset. For this reason, the audience is presumably academically inclined, and interested in self-improvement. The survey was posted to her account and received 10,563 responses within the first 36 h. Here we analyze the 4,789 of those responses that came from undergraduates. While we did not collect demographic or identifying information, we suspect that women are overrepresented in these data as followers of @unjadedjade are 80% women. A large minority of respondents were from the United Kingdom as Jade Bowler is a British influencer. Specifically, 43.3% of participants attend United Kingdom institutions, followed by 6.7% attending university in the Netherlands, 6.1% in Germany, 5.8% in the United States and 4.2% in Australia. Ninety additional countries are represented in these data (see Supplementary Figure 1 ).

Survey Design

The purpose of this survey is to learn about students’ instructional experiences following the transition to remote learning in the spring of 2020.

This survey was initially created for a student assignment for the undergraduate course Empirical Analysis at Minerva Schools at KGI. That version served as a robust pre-test and allowed for identification of the primary online platforms used, and the four primary modes of learning: synchronous (live) classes, recorded lectures and videos, uploaded or emailed materials, and chat-based communication. We did not adapt any open-ended questions based on the pre-test survey to avoid biasing the results and only corrected language in questions for clarity. We used these data along with an analysis of common practices in online learning to revise the survey. Our revised survey asked students to identify the synchronous and asynchronous pedagogical methods and platforms that they were using for remote learning. Pedagogical methods were drawn from literature assessing active and passive teaching strategies in North American institutions ( Fink, 2013 ; Chen et al., 2018 ; Davis et al., 2018 ). Open-ended questions asked students to describe why they preferred certain modes of learning and how they could improve their learning experience. Students also reported on their affective response to learning and participation using a Likert scale.

The revised survey also asked whether students had responded to the earlier survey. No significant differences were found between responses of those answering for the first and second times (data not shown). See Supplementary Appendix 1 for survey questions. Survey data was collected from 5/21/20 to 5/23/20.

Qualitative Coding

We applied a qualitative coding framework adapted from Gale et al. (2013) to analyze student responses to open-ended questions. Four researchers read several hundred responses and noted themes that surfaced. We then developed a list of themes inductively from the survey data and deductively from the literature on pedagogical practice ( Garrison et al., 1999 ; Zull, 2002 ; Fink, 2013 ; Freeman et al., 2014 ). The initial codebook was revised collaboratively based on feedback from researchers after coding 20–80 qualitative comments each. Before coding their assigned questions, alignment was examined through coding of 20 additional responses. Researchers aligned in identifying the same major themes. Discrepancies in terms identified were resolved through discussion. Researchers continued to meet weekly to discuss progress and alignment. The majority of responses were coded by a single researcher using the final codebook ( Supplementary Table 1 ). All responses to questions 3 (4,318 responses) and 8 (4,704 responses), and 2,512 of 4,776 responses to question 12 were analyzed. Valence was also indicated where necessary (i.e., positive or negative discussion of terms). This paper focuses on the most prevalent themes from our initial analysis of the qualitative responses. The corresponding author reviewed codes to ensure consistency and accuracy of reported data.

Statistical Analysis

The survey included two sets of Likert-scale questions, one consisting of a set of six statements about students’ perceptions of their experiences following the transition to remote learning ( Table 1 ). For each statement, students indicated their level of agreement with the statement on a five-point scale ranging from 1 (“Strongly Disagree”) to 5 (“Strongly Agree”). The second set asked the students to respond to the same set of statements, but about their retroactive perceptions of their experiences with in-person instruction before the transition to remote learning. This set was not the subject of our analysis but is present in the published survey results. To explore correlations among student responses, we used CrossCat analysis to calculate the probability of dependence between Likert-scale responses ( Mansinghka et al., 2016 ).

Table 1. Likert-scale questions.

Mean values are calculated based on the numerical scores associated with each response. Measures of statistical significance for comparisons between different subgroups of respondents were calculated using a two-sided Mann-Whitney U -test, and p -values reported here are based on this test statistic. We report effect sizes in pairwise comparisons using the common-language effect size, f , which is the probability that the response from a random sample from subgroup 1 is greater than the response from a random sample from subgroup 2. We also examined the effects of different modes of remote learning and technological platforms using ordinal logistic regression. With the exception of the mean values, all of these analyses treat Likert-scale responses as ordinal-scale, rather than interval-scale data.

Students Prefer Synchronous Class Sessions

Students were asked to identify their primary mode of learning given four categories of remote course design that emerged from the pilot survey and across literature on online teaching: live (synchronous) classes, recorded lectures and videos, emailed or uploaded materials, and chats and discussion forums. While 42.7% ( n = 2,045) students identified live classes as their primary mode of learning, 54.6% ( n = 2613) students preferred this mode ( Figure 1 ). Both recorded lectures and live classes were preferred over uploaded materials (6.22%, n = 298) and chat (3.36%, n = 161).

Figure 1. Actual (A) and preferred (B) primary modes of learning.

In addition to a preference for live classes, students whose primary mode was synchronous were more likely to enjoy the class, feel motivated and engaged, be satisfied with instruction and report higher levels of participation ( Table 2 and Supplementary Figure 2 ). Regardless of primary mode, over two-thirds of students reported they are often distracted during remote courses.

Table 2. The effect of synchronous vs. asynchronous primary modes of learning on student perceptions.

Variation in Pedagogical Techniques for Synchronous Classes Results in More Positive Perceptions of the Student Learning Experience

To survey the use of passive vs. active instructional methods, students reported the pedagogical techniques used in their live classes. Among the synchronous methods, we identify three different categories ( National Research Council, 2000 ; Freeman et al., 2014 ). Passive methods (P) include lectures, presentations, and explanation using diagrams, white boards and/or other media. These methods all rely on instructor delivery rather than student participation. Our next category represents active learning through primarily one-on-one interactions (A). The methods in this group are in-class assessment, question-and-answer (Q&A), and classroom chat. Group interactions (F) included classroom discussions and small-group activities. Given these categories, Mann-Whitney U pairwise comparisons between the 7 possible combinations and Likert scale responses about student experience showed that the use of a variety of methods resulted in higher ratings of experience vs. the use of a single method whether or not that single method was active or passive ( Table 3 ). Indeed, students whose classes used methods from each category (PAF) had higher ratings of enjoyment, motivation, and satisfaction with instruction than those who only chose any single method ( p < 0.0001) and also rated higher rates of participation and engagement compared to students whose only method was passive (P) or active through one-on-one interactions (A) ( p < 0.00001). Student ratings of distraction were not significantly different for any comparison. Given that sets of Likert responses often appeared significant together in these comparisons, we ran a CrossCat analysis to look at the probability of dependence across Likert responses. Responses have a high probability of dependence on each other, limiting what we can claim about any discrete response ( Supplementary Figure 3 ).

Table 3. Comparison of combinations of synchronous methods on student perceptions. Effect size (f).

Mann-Whitney U pairwise comparisons were also used to check if improvement in student experience was associated with the number of methods used vs. the variety of types of methods. For every comparison, we found that more methods resulted in higher scores on all Likert measures except distraction ( Table 4 ). Even comparison between four or fewer methods and greater than four methods resulted in a 59% chance that the latter enjoyed the courses more ( p < 0.00001) and 60% chance that they felt more motivated to learn ( p < 0.00001). Students who selected more than four methods ( n = 417) were also 65.1% ( p < 0.00001), 62.9% ( p < 0.00001) and 64.3% ( p < 0.00001) more satisfied with instruction, engaged, and actively participating, respectfully. Therefore, there was an overlap between how the number and variety of methods influenced students’ experiences. Since the number of techniques per category is 2–3, we cannot fully disentangle the effect of number vs. variety. Pairwise comparisons to look at subsets of data with 2–3 methods from a single group vs. 2–3 methods across groups controlled for this but had low sample numbers in most groups and resulted in no significant findings (data not shown). Therefore, from the data we have in our survey, there seems to be an interdependence between number and variety of methods on students’ learning experiences.

Table 4. Comparison of the number of synchronous methods on student perceptions. Effect size (f).

Variation in Asynchronous Pedagogical Techniques Results in More Positive Perceptions of the Student Learning Experience

Along with synchronous pedagogical methods, students reported the asynchronous methods that were used for their classes. We divided these methods into three main categories and conducted pairwise comparisons. Learning methods include video lectures, video content, and posted study materials. Interacting methods include discussion/chat forums, live office hours, and email Q&A with professors. Testing methods include assignments and exams. Our results again show the importance of variety in students’ perceptions ( Table 5 ). For example, compared to providing learning materials only, providing learning materials, interaction, and testing improved enjoyment ( f = 0.546, p < 0.001), motivation ( f = 0.553, p < 0.0001), satisfaction with instruction ( f = 0.596, p < 0.00001), engagement ( f = 0.572, p < 0.00001) and active participation ( f = 0.563, p < 0.00001) (row 6). Similarly, compared to just being interactive with conversations, the combination of all three methods improved five out of six indicators, except for distraction in class (row 11).

Table 5. Comparison of combinations of asynchronous methods on student perceptions. Effect size (f).

Ordinal logistic regression was used to assess the likelihood that the platforms students used predicted student perceptions ( Supplementary Table 2 ). Platform choices were based on the answers to open-ended questions in the pre-test survey. The synchronous and asynchronous methods used were consistently more predictive of Likert responses than the specific platforms. Likewise, distraction continued to be our outlier with no differences across methods or platforms.

Students Prefer In-Person and Synchronous Online Learning Largely Due to Social-Emotional Reasoning

As expected, 86.1% (4,123) of survey participants report a preference for in-person courses, while 13.9% (666) prefer online courses. When asked to explain the reasons for their preference, students who prefer in-person courses most often mention the importance of social interaction (693 mentions), engagement (639 mentions), and motivation (440 mentions). These students are also more likely to mention a preference for a fixed schedule (185 mentions) vs. a flexible schedule (2 mentions).

In addition to identifying social reasons for their preference for in-person learning, students’ suggestions for improvements in online learning focus primarily on increasing interaction and engagement, with 845 mentions of live classes, 685 mentions of interaction, 126 calls for increased participation and calls for changes related to these topics such as, “Smaller teaching groups for live sessions so that everyone is encouraged to talk as some people don’t say anything and don’t participate in group work,” and “Make it less of the professor reading the pdf that was given to us and more interaction.”

Students who prefer online learning primarily identify independence and flexibility (214 mentions) and reasons related to anxiety and discomfort in in-person settings (41 mentions). Anxiety was only mentioned 12 times in the much larger group that prefers in-person learning.

The preference for synchronous vs. asynchronous modes of learning follows similar trends ( Table 6 ). Students who prefer live classes mention engagement and interaction most often while those who prefer recorded lectures mention flexibility.

Table 6. Most prevalent themes for students based on their preferred mode of remote learning.

Student Perceptions Align With Research on Active Learning

The first, and most robust, conclusion is that incorporation of active-learning methods correlates with more positive student perceptions of affect and engagement. We can see this clearly in the substantial differences on a number of measures, where students whose classes used only passive-learning techniques reported lower levels of engagement, satisfaction, participation, and motivation when compared with students whose classes incorporated at least some active-learning elements. This result is consistent with prior research on the value of active learning ( Freeman et al., 2014 ).

Though research shows that student learning improves in active learning classes, on campus, student perceptions of their learning, enjoyment, and satisfaction with instruction are often lower in active-learning courses ( Deslauriers et al., 2019 ). Our finding that students rate enjoyment and satisfaction with instruction higher for active learning online suggests that the preference for passive lectures on campus relies on elements outside of the lecture itself. That might include the lecture hall environment, the social physical presence of peers, or normalization of passive lectures as the expected mode for on-campus classes. This implies that there may be more buy-in for active learning online vs. in-person.

A second result from our survey is that student perceptions of affect and engagement are associated with students experiencing a greater diversity of learning modalities. We see this in two different results. First, in addition to the fact that classes that include active learning outperform classes that rely solely on passive methods, we find that on all measures besides distraction, the highest student ratings are associated with a combination of active and passive methods. Second, we find that these higher scores are associated with classes that make use of a larger number of different methods.

This second result suggests that students benefit from classes that make use of multiple different techniques, possibly invoking a combination of passive and active methods. However, it is unclear from our data whether this effect is associated specifically with combining active and passive methods, or if it is associated simply with the use of multiple different methods, irrespective of whether those methods are active, passive, or some combination. The problem is that the number of methods used is confounded with the diversity of methods (e.g., it is impossible for a classroom using only one method to use both active and passive methods). In an attempt to address this question, we looked separately at the effect of number and diversity of methods while holding the other constant. Across a large number of such comparisons, we found few statistically significant differences, which may be a consequence of the fact that each comparison focused on a small subset of the data.

Thus, our data suggests that using a greater diversity of learning methods in the classroom may lead to better student outcomes. This is supported by research on student attention span which suggests varying delivery after 10–15 min to retain student’s attention ( Bradbury, 2016 ). It is likely that this is more relevant for online learning where students report high levels of distraction across methods, modalities, and platforms. Given that number and variety are key, and there are few passive learning methods, we can assume that some combination of methods that includes active learning improves student experience. However, it is not clear whether we should predict that this benefit would come simply from increasing the number of different methods used, or if there are benefits specific to combining particular methods. Disentangling these effects would be an interesting avenue for future research.

Students Value Social Presence in Remote Learning

Student responses across our open-ended survey questions show a striking difference in reasons for their preferences compared with traditional online learners who prefer flexibility ( Harris and Martin, 2012 ; Levitz, 2016 ). Students reasons for preferring in-person classes and synchronous remote classes emphasize the desire for social interaction and echo the research on the importance of social presence for learning in online courses.

Short et al. (1976) outlined Social Presence Theory in depicting students’ perceptions of each other as real in different means of telecommunications. These ideas translate directly to questions surrounding online education and pedagogy in regards to educational design in networked learning where connection across learners and instructors improves learning outcomes especially with “Human-Human interaction” ( Goodyear, 2002 , 2005 ; Tu, 2002 ). These ideas play heavily into asynchronous vs. synchronous learning, where Tu reports students having positive responses to both synchronous “real-time discussion in pleasantness, responsiveness and comfort with familiar topics” and real-time discussions edging out asynchronous computer-mediated communications in immediate replies and responsiveness. Tu’s research indicates that students perceive more interaction with synchronous mediums such as discussions because of immediacy which enhances social presence and support the use of active learning techniques ( Gunawardena, 1995 ; Tu, 2002 ). Thus, verbal immediacy and communities with face-to-face interactions, such as those in synchronous learning classrooms, lessen the psychological distance of communicators online and can simultaneously improve instructional satisfaction and reported learning ( Gunawardena and Zittle, 1997 ; Richardson and Swan, 2019 ; Shea et al., 2019 ). While synchronous learning may not be ideal for traditional online students and a subset of our participants, this research suggests that non-traditional online learners are more likely to appreciate the value of social presence.

Social presence also connects to the importance of social connections in learning. Too often, current systems of education emphasize course content in narrow ways that fail to embrace the full humanity of students and instructors ( Gay, 2000 ). With the COVID-19 pandemic leading to further social isolation for many students, the importance of social presence in courses, including live interactions that build social connections with classmates and with instructors, may be increased.

Limitations of These Data

Our undergraduate data consisted of 4,789 responses from 95 different countries, an unprecedented global scale for research on online learning. However, since respondents were followers of @unjadedjade who focuses on learning and wellness, these respondents may not represent the average student. Biases in survey responses are often limited by their recruitment techniques and our bias likely resulted in more robust and thoughtful responses to free-response questions and may have influenced the preference for synchronous classes. It is unlikely that it changed students reporting on remote learning pedagogical methods since those are out of student control.

Though we surveyed a global population, our design was rooted in literature assessing pedagogy in North American institutions. Therefore, our survey may not represent a global array of teaching practices.

This survey was sent out during the initial phase of emergency remote learning for most countries. This has two important implications. First, perceptions of remote learning may be clouded by complications of the pandemic which has increased social, mental, and financial stresses globally. Future research could disaggregate the impact of the pandemic from students’ learning experiences with a more detailed and holistic analysis of the impact of the pandemic on students.

Second, instructors, students and institutions were not able to fully prepare for effective remote education in terms of infrastructure, mentality, curriculum building, and pedagogy. Therefore, student experiences reflect this emergency transition. Single-modality courses may correlate with instructors who lacked the resources or time to learn or integrate more than one modality. Regardless, the main insights of this research align well with the science of teaching and learning and can be used to inform both education during future emergencies and course development for online programs that wish to attract traditional college students.

Global Student Voices Improve Our Understanding of the Experience of Emergency Remote Learning

Our survey shows that global student perspectives on remote learning agree with pedagogical best practices, breaking with the often-found negative reactions of students to these practices in traditional classrooms ( Shekhar et al., 2020 ). Our analysis of open-ended questions and preferences show that a majority of students prefer pedagogical approaches that promote both active learning and social interaction. These results can serve as a guide to instructors as they design online classes, especially for students whose first choice may be in-person learning. Indeed, with the near ubiquitous adoption of remote learning during the COVID-19 pandemic, remote learning may be the default for colleges during temporary emergencies. This has already been used at the K-12 level as snow days become virtual learning days ( Aspergren, 2020 ).

In addition to informing pedagogical decisions, the results of this survey can be used to inform future research. Although we survey a global population, our recruitment method selected for students who are English speakers, likely majority female, and have an interest in self-improvement. Repeating this study with a more diverse and representative sample of university students could improve the generalizability of our findings. While the use of a variety of pedagogical methods is better than a single method, more research is needed to determine what the optimal combinations and implementations are for courses in different disciplines. Though we identified social presence as the major trend in student responses, the over 12,000 open-ended responses from students could be analyzed in greater detail to gain a more nuanced understanding of student preferences and suggestions for improvement. Likewise, outliers could shed light on the diversity of student perspectives that we may encounter in our own classrooms. Beyond this, our findings can inform research that collects demographic data and/or measures learning outcomes to understand the impact of remote learning on different populations.

Importantly, this paper focuses on a subset of responses from the full data set which includes 10,563 students from secondary school, undergraduate, graduate, or professional school and additional questions about in-person learning. Our full data set is available here for anyone to download for continued exploration: https://dataverse.harvard.edu/dataset.xhtml?persistentId= doi: 10.7910/DVN/2TGOPH .

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

GS: project lead, survey design, qualitative coding, writing, review, and editing. TN: data analysis, writing, review, and editing. CN and PB: qualitative coding. JW: data analysis, writing, and editing. CS: writing, review, and editing. EV and KL: original survey design and qualitative coding. PP: data analysis. JB: original survey design and survey distribution. HH: data analysis. MP: writing. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We want to thank Minerva Schools at KGI for providing funding for summer undergraduate research internships. We also want to thank Josh Fost and Christopher V. H.-H. Chen for discussion that helped shape this project.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2021.647986/full#supplementary-material

Aspergren, E. (2020). Snow Days Canceled Because of COVID-19 Online School? Not in These School Districts.sec. Education. USA Today. Available online at: https://www.usatoday.com/story/news/education/2020/12/15/covid-school-canceled-snow-day-online-learning/3905780001/ (accessed December 15, 2020).

Google Scholar

Bostock, S. J. (1998). Constructivism in mass higher education: a case study. Br. J. Educ. Technol. 29, 225–240. doi: 10.1111/1467-8535.00066

CrossRef Full Text | Google Scholar

Bradbury, N. A. (2016). Attention span during lectures: 8 seconds, 10 minutes, or more? Adv. Physiol. Educ. 40, 509–513. doi: 10.1152/advan.00109.2016

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, B., Bastedo, K., and Howard, W. (2018). Exploring best practices for online STEM courses: active learning, interaction & assessment design. Online Learn. 22, 59–75. doi: 10.24059/olj.v22i2.1369

Davis, D., Chen, G., Hauff, C., and Houben, G.-J. (2018). Activating learning at scale: a review of innovations in online learning strategies. Comput. Educ. 125, 327–344. doi: 10.1016/j.compedu.2018.05.019

Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., and Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proc. Natl. Acad. Sci. 116, 19251–19257. doi: 10.1073/pnas.1821936116

Fink, L. D. (2013). Creating Significant Learning Experiences: An Integrated Approach to Designing College Courses. Somerset, NJ: John Wiley & Sons, Incorporated.

Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., et al. (2014). Active learning increases student performance in science, engineering, and mathematics. Proc. Natl. Acad. Sci. 111, 8410–8415. doi: 10.1073/pnas.1319030111

Gale, N. K., Heath, G., Cameron, E., Rashid, S., and Redwood, S. (2013). Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med. Res. Methodol. 13:117. doi: 10.1186/1471-2288-13-117

Garrison, D. R., Anderson, T., and Archer, W. (1999). Critical inquiry in a text-based environment: computer conferencing in higher education. Internet High. Educ. 2, 87–105. doi: 10.1016/S1096-7516(00)00016-6

Gay, G. (2000). Culturally Responsive Teaching: Theory, Research, and Practice. Multicultural Education Series. New York, NY: Teachers College Press.

Gillingham, and Molinari, C. (2012). Online courses: student preferences survey. Internet Learn. 1, 36–45. doi: 10.18278/il.1.1.4

Gillis, A., and Krull, L. M. (2020). COVID-19 remote learning transition in spring 2020: class structures, student perceptions, and inequality in college courses. Teach. Sociol. 48, 283–299. doi: 10.1177/0092055X20954263

Goodyear, P. (2002). “Psychological foundations for networked learning,” in Networked Learning: Perspectives and Issues. Computer Supported Cooperative Work , eds C. Steeples and C. Jones (London: Springer), 49–75. doi: 10.1007/978-1-4471-0181-9_4

Goodyear, P. (2005). Educational design and networked learning: patterns, pattern languages and design practice. Australas. J. Educ. Technol. 21, 82–101. doi: 10.14742/ajet.1344

Gunawardena, C. N. (1995). Social presence theory and implications for interaction and collaborative learning in computer conferences. Int. J. Educ. Telecommun. 1, 147–166.

Gunawardena, C. N., and Zittle, F. J. (1997). Social presence as a predictor of satisfaction within a computer mediated conferencing environment. Am. J. Distance Educ. 11, 8–26. doi: 10.1080/08923649709526970

Harris, H. S., and Martin, E. (2012). Student motivations for choosing online classes. Int. J. Scholarsh. Teach. Learn. 6, 1–8. doi: 10.20429/ijsotl.2012.060211

Levitz, R. N. (2016). 2015-16 National Online Learners Satisfaction and Priorities Report. Cedar Rapids: Ruffalo Noel Levitz, 12.

Mansinghka, V., Shafto, P., Jonas, E., Petschulat, C., Gasner, M., and Tenenbaum, J. B. (2016). CrossCat: a fully Bayesian nonparametric method for analyzing heterogeneous, high dimensional data. J. Mach. Learn. Res. 17, 1–49. doi: 10.1007/978-0-387-69765-9_7

National Research Council (2000). How People Learn: Brain, Mind, Experience, and School: Expanded Edition. Washington, DC: National Academies Press, doi: 10.17226/9853

Richardson, J. C., and Swan, K. (2019). Examining social presence in online courses in relation to students’ perceived learning and satisfaction. Online Learn. 7, 68–88. doi: 10.24059/olj.v7i1.1864

Shea, P., Pickett, A. M., and Pelz, W. E. (2019). A Follow-up investigation of ‘teaching presence’ in the suny learning network. Online Learn. 7, 73–75. doi: 10.24059/olj.v7i2.1856

Shekhar, P., Borrego, M., DeMonbrun, M., Finelli, C., Crockett, C., and Nguyen, K. (2020). Negative student response to active learning in STEM classrooms: a systematic review of underlying reasons. J. Coll. Sci. Teach. 49, 45–54.

Short, J., Williams, E., and Christie, B. (1976). The Social Psychology of Telecommunications. London: John Wiley & Sons.

Tu, C.-H. (2002). The measurement of social presence in an online learning environment. Int. J. E Learn. 1, 34–45. doi: 10.17471/2499-4324/421

Zull, J. E. (2002). The Art of Changing the Brain: Enriching Teaching by Exploring the Biology of Learning , 1st Edn. Sterling, VA: Stylus Publishing.

Keywords : online learning, COVID-19, active learning, higher education, pedagogy, survey, international

Citation: Nguyen T, Netto CLM, Wilkins JF, Bröker P, Vargas EE, Sealfon CD, Puthipiroj P, Li KS, Bowler JE, Hinson HR, Pujar M and Stein GM (2021) Insights Into Students’ Experiences and Perceptions of Remote Learning Methods: From the COVID-19 Pandemic to Best Practice for the Future. Front. Educ. 6:647986. doi: 10.3389/feduc.2021.647986

Received: 30 December 2020; Accepted: 09 March 2021; Published: 09 April 2021.

Reviewed by:

Copyright © 2021 Nguyen, Netto, Wilkins, Bröker, Vargas, Sealfon, Puthipiroj, Li, Bowler, Hinson, Pujar and Stein. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Geneva M. Stein, [email protected]

This article is part of the Research Topic

Covid-19 and Beyond: From (Forced) Remote Teaching and Learning to ‘The New Normal’ in Higher Education

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 09 May 2024

Looking back to move forward: comparison of instructors’ and undergraduates’ retrospection on the effectiveness of online learning using the nine-outcome influencing factors

• Yujie Su   ORCID: orcid.org/0000-0003-1444-1598 1 ,
• Xiaoshu Xu   ORCID: orcid.org/0000-0002-0667-4511 1 ,
• Yunfeng Zhang 2 ,
• Xinyu Xu 1 &
• Shanshan Hao 3  

Humanities and Social Sciences Communications volume  11 , Article number:  594 ( 2024 ) Cite this article

156 Accesses

Metrics details

• Language and linguistics

This study delves into the retrospections of undergraduate students concerning their online learning experiences after the COVID-19 pandemic, using the nine key influencing factors: behavioral intention, instruction, engagement, interaction, motivation, self-efficacy, performance, satisfaction, and self-regulation. 46 Year 1 students from a comprehensive university in China were asked to maintain reflective diaries throughout an academic semester, providing first-person perspectives on the strengths and weaknesses of online learning. Meanwhile, 18 college teachers were interviewed with the same questions as the students. Using thematic analysis, the research identified 9 factors. The research revealed that instruction ranked highest among the 9 factors, followed by engagement, self-regulation, interaction, motivation, and others. Moreover, teachers and students had different attitudes toward instruction. Thirdly, teacher participants were different from student participants given self-efficacy and self-regulation due to their variant roles in online instruction. Lastly, the study reflected students were not independent learners, which explained why instruction ranked highest in their point of view. Findings offer valuable insights for educators, administrators, and policy-makers involved in higher education. Recommendations for future research include incorporating a more diverse sample, exploring relationships between the nine factors, and focusing on equipping students with skills for optimal online learning experiences.

Similar content being viewed by others

Determinants of behaviour and their efficacy as targets of behavioural change interventions

Sleep quality, duration, and consistency are associated with better academic performance in college students

Interviews in the social sciences

Introduction.

The outbreak of the COVID-19 pandemic has had a profound impact on education worldwide, leading to the widescale adoption of online learning. According to the United Nations Educational, Scientific and Cultural Organization (UNESCO), at the peak of the pandemic, 192 countries had implemented nationwide closures, affecting approximately 99% of the world’s student population (UNESCO 2020 a). In response, educational institutions, teachers, and students quickly adapted to online learning platforms, leveraging digital technologies to continue education amidst the crisis (Marinoni et al. 2020 ).

The rapid and unexpected shift to online learning brought about a surge in research aiming to understand its impact, effectiveness, and challenges. Researchers across the globe have been investigating various dimensions of online learning. Some focus on students’ experiences and perspectives (Aristovnik et al. 2021 ), technological aspects (Bao 2020 ), pedagogical strategies (Hodges et al. 2020 ), and the socio-emotional aspect of learning (Ali 2020 ). Tan et al. ( 2021 ) found that motivation and satisfaction were mostly positively perceived by students, and lack of interaction was perceived as an unfavorable online instruction perception. Some center on teachers’ perceptions of the benefits and challenges (Lucas and Vicente, 2023 ; Mulla et al. 2023 ), post-pandemic pedagogisation (Rapanta et al. 2021 ), and post-pandemic further education (Kohnke et al. 2023 ; Torsani et al. 2023 ). It was worth noting that elements like interaction and engagement were central to the development and maintenance of the learning community (Lucas and Vincente 2023 ),

The rise of online learning has also posed unprecedented challenges. Studies have pointed out the digital divide and accessibility issues (Crawford et al. 2020 ), students’ motivation and engagement concerns (Martin and Bolliger 2018 ), and the need for effective online instructional practices (Trust and Whalen 2020 ). The rapid transition to online learning has highlighted the need for robust research to address these challenges and understand the effectiveness of online learning in this new educational paradigm.

Despite the extensive research on online learning during and after the COVID-19 pandemic, there remains a notable gap in understanding the retrospective perspectives of both undergraduates and teachers. Much of the current literature has focused on immediate response strategies to the transition to online learning, often overlooking the detailed insights that reflective retrospection can provide (Marinoni et al. 2020 ; Bao 2020 ). In addition, while many studies have examined isolated aspects of online learning, they have not often employed a comprehensive framework, leaving undergraduates’ voices, in particular, underrepresented in the discourse (Aristovnik et al. 2021 ; Crawford et al. 2020 ). This study, situated in the context of the COVID-19 pandemic’s impetus toward online learning, seeks to fill this crucial gap. By exploring online learning from the perspectives of both instructors and undergraduates, and analyzing nine key factors that include engagement, motivation, and self-efficacy, the research contributes vital insights into the dynamics of online education (Wang and Wang 2021 ). This exploration is especially pertinent as digital learning environments become increasingly prevalent worldwide (UNESCO 2020b ). The findings of our study are pivotal for shaping future educational policies and enhancing online education strategies in this continuously evolving educational landscape (Greenhow et al. 2021 ). Thus, three research questions were raised:

Q1: How do undergraduates and teachers in China retrospectively perceive the effectiveness of online learning after the COVID-19 pandemic?

Q2: Which of the nine outcome influencing factors had the most significant impact on online learning experiences after the pandemic, and why?

Q3: What recommendations can be proposed to enhance the effectiveness of online learning in the future?

The research takes place at a comprehensive university in China, with a sample of 46 Year 1 students and 18 experienced teachers. Their reflections on the effectiveness of online learning were captured through reflective diaries guided by four questions. These questions investigated the students’ online learning states and attitudes, identified issues and insufficiencies in online learning, analyzed the reasons behind these problems, and proposed improvements. By assessing their experiences and perceptions, we seek to explore the significant factors that shaped online learning outcomes after the pandemic and the means to enhance its effectiveness.

This paper first presents a review of the existing literature, focusing on the impact of the pandemic on online learning and discussing the nine significant factors influencing online learning outcomes. Following this, the methodology utilized for this study is detailed, setting the stage for a deeper understanding of the research process. Subsequently, we delve into the results of the thematic analysis conducted based on undergraduate students and teachers’ retrospections. Finally, the paper concludes by offering meaningful implications of the findings for various stakeholders and suggesting directions for future research in this critical area.

Literature review

Online learning application and evaluation in higher education.

Online learning, also known as e-learning or distance learning, refers to education that takes place over the Internet rather than in a traditional classroom setting. It has seen substantial growth over the past decade and has been accelerated due to the COVID-19 pandemic (Trust and Whalen 2020 ). Online learning allows for a flexible learning environment, breaking the temporal and spatial boundaries of traditional classroom settings (Bozkurt and Sharma 2020 ). In response to the COVID-19 pandemic, educational institutions globally have embraced online learning at an unprecedented scale. This has led to an immense surge in research focusing on the effects of the pandemic on online learning (Crawford et al. 2020 ; Marinoni et al. 2020 ).

Researchers were divided in their attitudes toward the effects of online learning, including positive, neutral, and negative. Research by Bahasoan et al. ( 2020 ), Bernard et al. ( 2004 ), Hernández-Lara and Serradell-López ( 2018 ), and Paechter and Maier ( 2010 ) indicated the effectiveness of online learning, including improved outcomes and engagement in online formats, providing flexibility and enhancing digital skills for instance. Research, including studies by Dolan Hancock and Wareing ( 2015 ) and Means et al. ( 2010 ), indicates that under equivalent conditions and with similar levels of support, there is frequently no substantial difference in learning outcomes between traditional face-to-face courses and completely online courses.

However, online learning was not without its challenges. Research showing less favorable results for specific student groups can be referenced in Dennen ( 2014 ), etc. The common problems faced by students included underdeveloped independent learning ability, lack of motivation, difficulties in self-regulation, student engagement and technical issues (Aristovnik et al. 2021 ; Martin and Bolliger 2018 ; Song et al. 2004 ; Zheng et al. 2022 ).

Moreover, factors like instructional strategies, course design, etc. were also linked to learning outcomes and successful online learning (Ali 2020 ; Hongsuchon et al. 2022 ). Careaga-Butter et al. ( 2020 ) critically analyze online education in pandemic and post-pandemic contexts, focusing on digital tools and resources for teaching in synchronous and asynchronous learning modalities. They discuss the swift adaptation to online learning during the pandemic, highlighting the importance of technological infrastructure, pedagogical strategies, and the challenges of digital divides. The article emphasizes the need for effective online learning environments and explores trends in post-pandemic education, providing insights into future educational strategies and practices.

Determinants of online learning outcomes

Online learning outcomes in this paper refer to the measurable educational results achieved through online learning methods, including knowledge acquisition, skill development, changes in attitudes or behaviors, and performance improvements (Chang 2016 ; Panigrahi et al. 2018 ). The literature review identified key factors influencing online learning outcomes, emphasizing their significant role in academic discourse. These factors, highlighted in scholarly literature, include student engagement, instructional design, technology infrastructure, student-teacher interaction, and student self-regulation.

Student Engagement: The level of a student’s engagement significantly impacts their learning outcomes. The more actively a student is engaged with the course content and activities, the better their performance tends to be. This underscores the importance of designing engaging course content and providing opportunities for active learning in an online environment (Martin and Bolliger 2018 ).

Instructional Design: How an online course is designed can greatly affect student outcomes. Key elements such as clarity of learning objectives, organization of course materials, and the use of diverse instructional strategies significantly impact student learning (Bozkurt and Sharma 2020 ).

Technology Infrastructure: The reliability and ease of use of the learning management system (LMS) also play a significant role in online learning outcomes. When students experience technical difficulties, it can lead to frustration, reduced engagement, and lower performance (Johnson et al. 2020 ).

Student-Teacher Interaction: Interaction between students and teachers in an online learning environment is a key determinant of successful outcomes. Regular, substantive feedback from instructors can promote student learning and motivation (Boling et al. 2012 ).

Student Self-Regulation: The autonomous nature of online learning requires students to be proficient in self-regulated learning, which involves setting learning goals, self-monitoring, and self-evaluation. Students who exhibit strong self-regulation skills are more likely to succeed in online learning (Broadbent 2017 ).

While many studies have investigated individual factors affecting online learning, there is a paucity of research offering a holistic view of these factors and their interrelationships, leading to a fragmented understanding of the influences on online learning outcomes. Given the multitude of experiences and variables encompassed by online learning, a comprehensive framework like is instrumental in ensuring a thorough investigation and interpretation of the breadth of students’ experiences.

Students’ perceptions of online learning

Understanding students’ perceptions of online learning is essential for enhancing its effectiveness and student satisfaction. Studies show students appreciate online learning for its flexibility and convenience, offering personalized learning paths and resource access (Händel et al. 2020 ; Johnson et al. 2020 ). Yet, challenges persist, notably in maintaining motivation and handling technical issues (Aristovnik et al. 2021 ; Händel et al. 2020 ). Aguilera-Hermida ( 2020 ) reported mixed feelings among students during the COVID-19 pandemic, including feelings of isolation and difficulty adjusting to online environments. Boling et al. ( 2012 ) emphasized students’ preferences for interactive and communicative online learning environments. Additionally, research indicates that students seek more engaging content and innovative teaching approaches, suggesting a gap between current online offerings and student expectations (Chakraborty and Muyia Nafukho 2014 ). Students also emphasize the importance of community and peer support in online settings, underlining the need for collaborative and social learning opportunities (Lai et al. 2019 ). These findings imply that while online learning offers significant benefits, addressing its shortcomings is critical for maximizing its potential.

The pandemic prompted a reconsideration of instructional modalities, with many students favoring face-to-face instruction due to the immediacy and focus issues (Aristovnik et al. 2021 ; Trust and Whalen 2020 ). Despite valuable insights, research gaps remain, particularly in long-term undergraduate reflections and the application of nine factors of comprehensive frameworks, indicating a need for more holistic research in online learning effectiveness.

Teachers’ perceptions of online learning

The pandemic has brought attention to how teachers manage instruction in virtual learning environments. Teachers and students are divided in terms of their attitudes toward online learning. Some teachers and students looked to the convenience and flexibility of online learning (Chuenyindee et al. 2022 ; Al-Emran and Shaalan 2021 ). They conceived that online learning provided opportunities to improve educational equality as well (Tenório et al. 2016 ). Even when COVID-19 was over, the dependence on online learning was likely here to stay, for some approaches of online learning were well-received by students and teachers (Al-Rahmi et al. 2019 ; Hongsuchon et al. 2022 ).

Teachers had shown great confidence in delivering instruction in an online environment in a satisfying manner. They also agreed that the difficulty of teaching was closely associated with course structures (Gavranović and Prodanović 2021 ).

Not all were optimistic about the effects of online learning. They sought out the challenges facing teachers and students during online learning.

A mixed-method study of K-12 teachers’ feelings, experiences, and perspectives that the major challenges faced by teachers during the COVID-19 pandemic were lack of student participation and engagement, technological support for online learning, lack of face-to-face interactions with students, no work-life balance and learning new technology.

The challenges to teachers’ online instruction included instruction technology (Maatuk et al. 2022 ; Rasheed et al. 2020 ), course design (Khojasteh et al. 2023 ), and teachers’ confidence (Gavranović and Prodanović 2021 ).

Self-regulation challenges and challenges in using technology were the key challenges to students, while the use of technology for teaching was the challenge facing teachers (Rasheed et al. 2020 ).

The quality of course design was another important factor in online learning. A research revealed the competency of the instructors and their expertise in content development contributed a lot to students’ satisfaction with the quality of e-contents.

Theoretical framework

The theoretical foundation of the research is deeply rooted in multifaceted framework for online learning, which provides a comprehensive and interwoven model encompassing nine critical factors that collectively shape the educational experience in online settings. This framework is instrumental in guiding our analysis and enhances the comparability and interpretability of our results within the context of existing literature.

Central to Yu’s framework is the concept of behavioral intention, which acts as a precursor to student engagement in online learning environments. This engagement, inherently linked to the students’ intentions and motivations, is significantly influenced by the quality of instruction they receive. Instruction, therefore, emerges as a pivotal element in this model, directly impacting not only student engagement but also fostering a sense of self-efficacy among learners. Such self-efficacy is crucial as it influences both the performance of students and their overall satisfaction with the learning process.

The framework posits that engagement, a derivative of both strong behavioral intention and effective instruction, plays a vital role in enhancing student performance. This engagement is tightly interlaced with self-regulation, an indispensable skill in the autonomous and often self-directed context of online learning. Interaction, encompassing various forms such as student-teacher and peer-to-peer communications, further enriches the learning experience. It significantly contributes to the development of motivation and self-efficacy, both of which are essential for sustaining engagement and fostering self-regulated learning.

Motivation, especially when intrinsically driven, acts as a catalyst, perpetuating engagement and self-regulation, which ultimately leads to increased satisfaction with the learning experience. In this framework, self-efficacy, nurtured through effective instruction and meaningful interactions, has a positive impact on students’ performance and satisfaction, thereby creating a reinforcing cycle of learning and achievement.

Performance in this model is viewed as a tangible measure of the synergistic interplay of engagement, instructional quality, and self-efficacy, while satisfaction reflects the culmination of the learning experience, shaped by the quality of instruction, the extent and nature of interactions, and the flexibility of the learning environment. This satisfaction, in turn, influences students’ future motivation and their continued engagement with online learning.

Yu’s model thus presents a dynamic ecosystem where changes in one factor can have ripple effects across the entire spectrum of online learning. It emphasizes the need for a holistic approach in the realm of online education, considering the complex interplay of these diverse yet interconnected elements to enhance both the effectiveness and the overall experience of online learning.

The current study employed a qualitative design to explore teachers’ and undergraduates’ retrospections on the effectiveness of online learning during the first semester of the 2022–2023 school year, which is in the post-pandemic period. Data were collected using reflective diaries, and thematic analysis was applied to understand the experiences based on the nine factors.

Sample and sampling

The study involved 18 teachers and 46 first-year students from a comprehensive university in China, selected through convenience sampling to ensure diverse representation across academic disciplines. To ensure a diverse range of experiences in online learning, the participant selection process involved an initial email inquiry about their prior engagement with online education. The first author of this study received ethics approval from the department research committee, and participants were informed of the study’s objectives two weeks before via email. Only those participants who provided written informed consent were included in the study and were free to withdraw at any time. Pseudonyms were used to protect participants’ identities during the data-coding process. For direct citations, acronyms of students’ names were used, while “T+number” was used for citations from teacher participants.

The 46 students are all first-year undergraduates, 9 females and 37 males majoring in English and non-English (see Table 1 ).

The 18 teachers are all experienced instructors with at least 5 years of teaching experience, 13 females and 5 male, majoring in English and Non-English (see Table 2 ).

Data collection

Students’ data were collected through reflective diaries in class during the first semester of the 2022–2023 school year. Each participant was asked to maintain a diary over the course of one academic semester, in which they responded to four questions.

The four questions include:

What was your state and attitude toward online learning?

What were the problems and shortcomings of online learning?

What do you think are the reasons for these problems?

What measures do you think should be taken to improve online learning?

This approach provided a first-person perspective on the participants’ online teaching or learning experiences, capturing the depth and complexity of their retrospections.

Teachers were interviewed separately by responding to the four questions the same as the students. Each interview was conducted in the office or the school canteen during the semester and lasted about 20 to 30 min.

Data analysis

We utilized thematic analysis to interpret the reflective diaries, guided initially by nine factors. This method involved extensive engagement with the data, from initial coding to the final report. While Yu’s factors provided a foundational structure, we remained attentive to new themes, ensuring a comprehensive analysis. Our approach was methodical: familiarizing ourselves with the data, identifying initial codes, systematically searching and reviewing themes, and then defining and naming them. To validate our findings, we incorporated peer debriefing, and member checking, and maintained an audit trail. This analysis method was chosen for its effectiveness in extracting in-depth insights from undergraduates’ retrospections on their online learning experiences post-pandemic, aligning with our research objectives.

According to the nine factors, the interviews of 18 teachers and 46 Year 1 undergraduates were catalogued and listed in Table 3 .

Behavioral intention towards online learning post-pandemic

Since the widespread of the COVID-19 pandemic, both teachers and students have experienced online learning. However, their online teaching or learning was forced rather than planned (Baber 2021 ; Bao 2020 ). Students more easily accepted online learning when they perceived the severity of COVID-19.

When entering the post-pandemic era, traditional teaching was resumed. Students often compared online learning with traditional learning by mentioning learning interests, eye contact, face-to-face learning and learning atmosphere.

“I don’t think online learning is a good form of learning because it is hard to focus on learning.” (DSY) “In unimportant courses, I would let the computer log to the platform and at the same time do other entertains such as watching movies, listening to the music, having snacks or do the cleaning.” (XYN) “Online learning makes it impossible to have eye contact between teachers and students and unable to create a face-to-face instructional environment, which greatly influences students’ initiative and engagement in classes.” (WRX)

They noted that positive attitudes toward online learning usually generated higher behavioral intention to use online learning than those with negative attitudes, as found in the research of Zhu et al. ( 2023 ). So they put more blame on distractions in the learning environment.

“Online learning relies on computers or cell phones which easily brings many distractions. … I can’t focus on studying, shifting constantly from study and games.” (YX) “When we talk about learning online, we are hit by an idea that we can take a rest in class. It’s because everyone believes that during online classes, the teacher is unable to see or know what we are doing.” (YM) “…I am easily disturbed by external factors, and I am not very active in class.” (WZB)

Teachers reported a majority of students reluctantly turning on their cameras during online instruction and concluded the possible reason for such behavior.

“One of the reasons why some students are unwilling to turn on the camera is that they are worried about their looks and clothing at home, or that they don’t want to become the focus.” (T4)

They also noticed students’ absent-mindedness and lazy attitude during online instruction.

“As for some students who are not self-regulated, they would not take online learning as seriously as offline learning. Whenever they are logged onto the online platform, they would be unable to stay focused and keep their attention.” (T1)

Challenges and opportunities in online instruction post-pandemic

Online teaching brought new challenges and opportunities for students during and after the pandemic. The distractions at home seemed to be significantly underestimated by teachers in an online learning environment (Radmer and Goodchild 2021 ). It might be the reason why students greatly expected and heavily relied on teachers’ supervision and management.

“The biggest problem of online learning is that online courses are as imperative as traditional classes, but not managed face to face the same as the traditional ones.” (PC) “It is unable to provide some necessary supervision.” (GJX) “It is incapable of giving timely attention to every student.” (GYC) “Teachers can’t understand students’ conditions in time in most cases so teachers can’t adjust their teaching plan.” (MZY) “Some courses are unable to reach the teaching objectives due to lack of experimental conduction and practical operation.” (YZH) “Insufficient teacher-student interaction and the use of cell phones make both groups unable to engage in classes. What’s more, though online learning doesn’t put a high requirement for places, its instructional environment may be crucial due to the possible distractions.” (YCY)

Teachers also viewed online instruction as an addition to face-to-face instruction.

“Online learning cannot run as smoothly as face-to-face instruction, but it can provide an in-time supplement to the practical teaching and students’ self-learning.” (T13, T17) “Online instruction is an essential way to ensure the normal function of school work during the special periods like the pandemic” (T1, T15)

Factors influencing student engagement in online learning

Learning engagement was found to contribute to gains in the study (Paul and Diana 2006 ). It was also referred to as a state closely intertwined with the three dimensions of learning, i.e., vigor, dedication, and absorption (Schaufeli et al. 2002 ). Previous studies have found that some key factors like learning interaction, self-regulation, and social presence could influence learning engagement and learning outcomes (Lowenthal and Dunlap 2020 ; Ng 2018 ). Due to the absence of face-to-face interaction like eye contact, facial expressions and body language, both groups of interviewees agreed that the students felt it hard to keep their attention and thus remain active in online classes.

“Students are unable to engage in study due to a lack of practical learning environment of online learning.” (ZMH, T12) “Online platforms may not provide the same level of engagement and interaction as in-person classrooms, making it harder for students to ask questions or engage in discussions.” (HCK) “The Internet is cold, lack of emotional clues and practical connections, which makes it unable to reproduce face-to-face offline learning so that teachers and students are unlikely to know each other’s true feelings or thoughts. In addition, different from the real-time learning supervision in offline learning, online learning leaves students more learning autonomy.” (XGH) “Lack of teachers’ supervision and practical learning environment, students are easily distracted.” (LMA, T9)

Just as Zhu et al. ( 2023 ) pointed out, we had been too optimistic about students’ engagement in online learning, because online learning relied more on students’ autonomy and efforts to complete online learning.

Challenges in teacher-student interaction in online learning

Online learning has a notable feature, i.e., a spatial and temporal separation among teachers and students. Thus, online teacher-student interactions, fundamentals of relationship formation, have more challenges for both teachers and students. The prior studies found that online interaction affected social presence and indirectly affected learning engagement through social presence (Miao and Ma 2022 ). In the present investigation, both teachers and students noted the striking disadvantage of online interaction.

“Online learning has many problems such as indirect teacher-student communication, inactive informative communication, late response of students and their inability to reflect their problems. For example, teachers cannot evaluate correctly whether the students have mastered or not.” (YYN) “Teachers and students are separated by screens. The students cannot make prompt responses to the teachers’ questions via loudspeakers or headphones. It is not convenient for students to participate in questioning and answering. …for most of the time, the students interact with teachers via typing.” (ZJY) “While learning online, students prefer texting the questions to answering them via the loudspeaker.”(T7)

Online learning interaction was also found closely related to online learning engagement, performance, and self-efficacy.

“Teachers and students are unable to have timely and effective communication, which reduces the learning atmosphere. Students are often distracted. While doing homework, the students are unable to give feedback to teachers.” (YR) “Students are liable to be distracted by many other side matters so that they can keep their attention to online learning.” (T15)

In the online learning environment, teachers need to make efforts to build rapport and personalizing interactions with students to help them perform better and achieve greater academic success (Harper 2018 ; Ong and Quek 2023 ) Meanwhile, teachers should also motivate students’ learning by designing the lessons, giving lectures and managing the processes of student interactions (Garrison 2003 ; Ong and Quek 2023 ).

Determinants of self-efficacy in online learning

Online learning self-efficacy refers to students’ perception of their abilities to fulfill specific tasks required in online learning (Calaguas and Consunji 2022 ; Zimmerman and Kulikowich 2016 ). Online learning self-efficacy was found to be influenced by various factors including task, learner, course, and technology level, among which task level was found to be most closely related (Xu et al. 2022 ). The responses from the 46 student participants reveal a shared concern, albeit without mentioning specific tasks; they highlight critical aspects influencing online learning: learner attributes, course structure, and technological infrastructure.

One unifying theme from the student feedback is the challenge of self-regulation and environmental distractions impacting learning efficacy. For instance, participant WSX notes the necessity for students to enhance time management skills due to deficiencies in self-regulation, which is crucial for successful online learning. Participant WY expands on this by pointing out the distractions outside traditional classroom settings, coupled with limited teacher-student interaction, which hampers idea exchange and independent thought, thereby undermining educational outcomes. These insights suggest a need for strategies that bolster students’ self-discipline and interactive opportunities in virtual learning environments.

On the technological front, participants WT and YCY address different but related issues. Participant WT emphasizes the importance of up-to-date course content and learning facilities, indicating that outdated materials and tools can significantly diminish the effectiveness of online education. Participant YCY adds to this by highlighting problems with online learning applications, such as subpar functionalities that can introduce additional barriers to learning.

Teacher participants, on the other hand, shed light on objective factors predominantly related to course content and technology. Participant T5’s response underscores the heavy dependency on technological advancement in online education and points out the current inability of platforms or apps to adequately monitor student engagement and progress. Participant T9 voices concerns about course content not being updated or aligned with contemporary trends and student interests, suggesting a disconnect between educational offerings and learner needs. Meanwhile, participant T8 identifies unstable network services as a significant hindrance to online teaching, highlighting infrastructure as a critical component of online education’s success.

Teachers also believed the insufficient mastery of facilities and unfamiliarity with online instruction posed difficulty.

“Most teachers and students are not familiar with online instruction. For example, some teachers are unable to manage online courses so they cannot design the courses well. Some students lack self-regulation, which leads to their distraction or avoidance in class.” (T9)

Influences on student performance in online learning

Students’ performance during online lessons is closely associated with their satisfaction and self-efficacy. Most of the student participants reflected on their distractions, confusion, and needs, which indicates their dissatisfaction with online learning.

“During online instruction, it is convenient for the students to make use of cell phones, but instead, cell phones bring lots of distraction.” (YSC) “Due to the limits of online learning, teachers are facing the computer screen and unable to know timely students’ needs and confusion. Meanwhile, it’s inconvenient for teachers to make clear explanations of the sample questions or problems.” (HZW)

They thought their low learning efficiency in performance was caused by external factors like the learning environment.

“The most obvious disadvantage of online learning goes to low efficiency. Students find it hard to keep attention to study outside the practical classroom or in a relaxing environment.” (WY) “Teachers are not strict enough with students, which leads to ineffective learning.” (WRX)

Teacher participants conceived students’ performance as closely related to valid online supervision and students’ self-regulation.

“Online instruction is unable to create a learning environment, which helps teachers know students’ instant reaction. Only when students well regulate themselves and stay focused during online learning can they achieve successful interactions and make good accomplishments in the class.” (T11) “Some students need teachers’ supervision and high self-regulation, or they were easily distracted.” (T16)

Student satisfaction and teaching effectiveness in online learning

Online learning satisfaction was found to be significantly and positively associated with online learning self-efficacy (Al-Nasa’h et al. 2021 ; Lashley et al. 2022 ). Around 46% of student participants were unsatisfied with teachers’ ways of teaching.

“Comparatively, bloggers are more interesting than teachers’ boring and dull voices in online learning.” (DSY) “Teachers’ voice sounds dull and boring through the internet, which may cause listeners to feel sleepy, and the teaching content is not interesting enough to the students.” (MFE)

It reflected partly that some teachers were not adapted to online teaching possibly due to a lack in experience of online teaching or learning (Zhu et al. 2022 ).

“Some teachers are not well-prepared for online learning. They are particularly unready for emergent technological problems when delivering the teaching.” (T1) “One of the critical reasons lies in the fact that teachers and students are not well trained before online learning. In addition, the online platform is not unified by the college administration, which has led to chaos and difficulty of online instruction.” (T17)

Teachers recognized their inadequate preparation and mastery of online learning as one of the reasons for dissatisfaction, but student participants exaggerated the role of teachers in online learning and ignored their responsibility in planning and managing their learning behavior, as in the research of (Xu et al. 2022 ).

The role of self-regulation in online learning success

In the context of online learning, self-regulation stands out as a crucial factor, necessitating heightened levels of student self-discipline and autonomy. This aspect, as Zhu et al. ( 2023 ) suggest, grants students significant control over their learning processes, making it a vital component for successful online education.

“Online learning requires learners to be of high discipline and self-regulation. Without good self-regulation, they are less likely to be effective in online learning.” (YZJ) “Most students lack self-control, unable to control the time of using electronic products. Some even use other electronic products during online learning, which greatly reduces their efficiency in learning.” (GPY) “Students are not well developed in self-control and easily distracted. Thus they are unable to engage fully in their study, which makes them unable to keep up with others” (XYN)

Both groups of participants had a clear idea of the positive role of self-regulation in successful learning, but they also admitted that students need to strengthen their self-regulation skills and it seemed they associated with the learning environment, learning efficiency and teachers’ supervision.

“If they are self-motivated, online learning can be conducted more easily and more efficiently. However, a majority are not strong in regulating themselves. Teachers’ direct supervision in offline learning can do better in motivating them to study hard…lack of interaction makes students less active and motivated.” (LY) “Students have a low level of self-discipline. Without teachers’ supervision, they find it hard to listen attentively or even quit listening. Moreover, in class, the students seldom think actively and independently.” (T13)

The analysis of participant responses, categorized into three distinct attitude groups – positive, neutral, and negative – reveals a multifaceted view of the disadvantages of online learning, as shown in Tables 4 and 5 . This classification provides a clearer understanding of how attitudes towards online learning influence perceptions of self-regulation and other related factors.

In Table 4 , the division among students is most pronounced in terms of interaction and self-efficacy. Those with neutral attitudes highlighted interaction as a primary concern, suggesting that it is less effective in an online setting. Participants with positive attitudes noted a lack of student motivation, while those with negative views emphasized the need for better self-efficacy. Across all attitudes, instruction, engagement, self-regulation, and behavior intention were consistently identified as areas needing improvement.

Table 5 sheds light on teachers’ perspectives, revealing a consensus on the significance and challenges of instruction, motivation, and self-efficacy in online learning. Teachers’ opinions vary most significantly on self-efficacy and engagement. Those with negative attitudes point to self-efficacy and instructional quality as critical areas needing attention, while neutral attitudes focus on the role of motivation.

Discussions

Using a qualitative and quantitative analysis of the questionnaire data showed that among the 18 college teachers and 46 year 1 undergraduate students of various majors taking part in the interview, about 38.9% of teachers and about 30.4% of students supported online learning. Only two teachers were neutral about online learning, and 50% of teachers did not support virtual learning. The percentages of students who expressed positive and neutral views on online learning were the same, i.e., 34.8%. This indicates that online learning could serve as a complementary approach to traditional education, yet it is not without challenges, particularly in terms of student engagement, self-regulation, and behavioral intention, which were often attributed to distractions inherent in online environments.

In analyzing nine factors, it was evident that both teachers and students did not perceive these factors uniformly. Instruction was a significant element for both groups, as validated by findings in Tables 3 and 5 . The absence of face-to-face interactions in online learning shifted the focus to online instruction quality. Teachers cited technological challenges as a central concern, while students criticized the lack of engaging content and teaching methods. This aligns with Miao and Ma ( 2022 ), who argued that direct online interaction does not necessarily influence learner engagement, thus underscoring the need for integrated approaches encompassing interactions, self-regulation, and social presence.

Furthermore, the role of technology acceptance in shaping self-efficacy was highlighted by Xu et al. ( 2022 ), suggesting that students with higher self-efficacy tend to challenge themselves more. Chen and Hsu ( 2022 ) noted the positive influence of using emojis in online lessons, emphasizing the importance of innovative pedagogical approaches in online settings.

The study revealed distinct priorities between teachers and students in online learning: teachers emphasized effective instruction delivery, while students valued learning outcomes, self-regulation, and engagement. This divergence highlights the unique challenges each group faces. Findings by Dennen et al. ( 2007 ) corroborate this, showing instructors focusing on content and guidance, while students prioritize interpersonal communication and individualized attention. Additionally, Lee et al. ( 2011 ) found that reduced transactional distance and increased student engagement led to enhanced perceptions of learning outcomes, aligning with students’ priorities in online courses. Understanding these differing perspectives is crucial for developing comprehensive online learning strategies that address the needs of both educators and learners.

Integrating these findings with broader contextual elements such as technological infrastructure, pedagogical strategies, socio-economic backgrounds, and environmental factors (Balanskat and Bingimlas 2006 ) further enriches our understanding. The interplay between these external factors and Yu’s nine key aspects forms a complex educational ecosystem. For example, government interventions and training programs have been shown to increase teachers’ enthusiasm for ICT and its routine use in education (Balanskat and Bingimlas 2006 ). Additionally, socioeconomic factors significantly impact students’ experiences with online learning, as the digital divide in connectivity and access to computers at home influences the ICT experience, an important factor for school achievement (OECD 2015 ; Punie et al. 2006 ).

In sum, the study advocates for a holistic approach to understanding and enhancing online education, recognizing the complex interplay between internal factors and external elements that shape the educational ecosystem in the digital age.

Conclusion and future research

This study offered a comprehensive exploration into the retrospective perceptions of college teachers and undergraduate students regarding their experiences with online learning following the COVID-19 pandemic. It was guided by a framework encompassing nine key factors that influence online learning outcomes. To delve into these perspectives, the research focused on three pivotal questions. These questions aimed to uncover how both undergraduates and teachers in China view the effectiveness of online learning post-pandemic, identify which of the nine influencing factors had the most significant impact, and propose recommendations for enhancing the future effectiveness of online learning.

In addressing the first research question concerning the retrospective perceptions of online learning’s effectiveness among undergraduates and teachers in China post-COVID-19 pandemic, the thematic analysis has delineated clear divergences in attitude between the two demographics. Participants were primarily divided into three categories based on their stance toward online learning: positive, neutral, and negative. The results highlighted a pronounced variance in attitude distribution between teachers and students, with a higher percentage of teachers expressing clear-cut opinions, either favorably or unfavorably, towards the effectiveness of online learning.

Conversely, students displayed a pronounced inclination towards neutrality, revealing a more cautious or mixed stance on the effectiveness of online learning. This prevalent neutrality within the student body could be attributed to a range of underlying reasons. It might signify students’ uncertainties or varied experiences with online platforms, differences in engagement levels, gaps in digital literacy, or fluctuating quality of online materials and teaching methods. Moreover, this neutral attitude may arise from the psychological and social repercussions of the pandemic, which have potentially altered students’ approaches to and perceptions of learning in an online context.

In the exploration of the nine influential factors in online learning, it was discovered that both teachers and students overwhelmingly identified instruction as the most critical aspect. This was closely followed by engagement, interaction, motivation, and other factors, while performance and satisfaction were perceived as less influential by both groups. However, the attitudes of teachers and students towards these factors revealed notable differences, particularly about instruction. Teachers often attributed challenges in online instruction to technological issues, whereas students perceived the quality of instruction as a major influence on their learning effectiveness. This dichotomy highlights the distinct perspectives arising from their different roles within the educational process.

A further divergence was observed in views on self-efficacy and self-regulation. Teachers, with a focus on delivering content, emphasized the importance of self-efficacy, while students, grappling with the demands of online learning, prioritized self-regulation. This reflects their respective positions in the online learning environment, with teachers concerned about the efficacy of their instructional strategies and students about managing their learning process. Interestingly, the study also illuminated that students did not always perceive themselves as independent learners, which contributed to the high priority they placed on instruction quality. This insight underlines a significant area for development in online learning strategies, emphasizing the need for fostering greater learner autonomy.

Notably, both teachers and students concurred that stimulating interest was a key factor in enhancing online learning. They proposed innovative approaches such as emulating popular online personalities, enhancing interactive elements, and contextualizing content to make it more relatable to students’ lives. Additionally, practical suggestions like issuing preview tasks and conducting in-class quizzes were highlighted as methods to boost student engagement and learning efficiency. The consensus on the importance of supervisory roles underscores the necessity for a balanced approach that integrates guidance and independence in the online learning environment.

The outcomes of our study highlight the multifaceted nature of online learning, accentuated by the varied perspectives and distinct needs of teachers and students. This complexity underscores the necessity of recognizing and addressing these nuances when designing and implementing online learning strategies. Furthermore, our findings offer a comprehensive overview of both the strengths and weaknesses of online learning during an unprecedented time, offering valuable insights for educators, administrators, and policy-makers involved in higher education. Moreover, it emphasized the intricate interplay of multiple factors—behavioral intention, instruction, engagement, interaction, motivation, self-efficacy, performance, satisfaction, and self-regulation—in shaping online learning outcomes. presents some limitations, notably its reliance on a single research method and a limited sample size.

However, the exclusive use of reflective diaries and interviews restricts the range of data collection methods, which might have been enriched by incorporating additional quantitative or mixed-method approaches. Furthermore, the sample, consisting only of students and teachers from one university, may not adequately represent the diverse experiences and perceptions of online learning across different educational contexts. These limitations suggest the need for a cautious interpretation of the findings and indicate areas for future research expansion. Future research could extend this study by incorporating a larger, more diverse sample to gain a broader understanding of undergraduate students’ retrospections across different contexts and cultures. Furthermore, research could also explore how to better equip students with the skills and strategies necessary to optimize their online learning experiences, especially in terms of the self-regulated learning and motivation aspects.

Data availability

The data supporting this study is available from https://doi.org/10.6084/m9.figshare.25583664.v1 . The data consists of reflective diaries from 46 Year 1 students from a comprehensive university in China and 18 college teachers. We utilized thematic analysis to interpret the reflective diaries, guided initially by nine factors. The results highlight the critical need for tailored online learning strategies and provide insights into its advantages and challenges for stakeholders in higher education.

Aguilera-Hermida AP (2020) College students’ use and acceptance of emergency online learning due to COVID-19. Int. J. Educ. Res. Open 1:100011. https://doi.org/10.1016/j.ijedro.2020.100011

Article   Google Scholar  

Al-Emran, M, & Shaalan, K (2021, October 27). E-podium technology: A medium of managing knowledge at Al Buraimi University College via M-learning. In Proceedings of the 2nd BCS International IT Conference, Abu Dhabi, United Arab Emirates 2014. Retrieved October 17, 2023, from https://dblp.uni-trier.de/rec/conf/bcsit/EmranS14.html

Ali W (2020) Online and remote learning in higher education institutes: A necessity in light of COVID-19 pandemic. High. Educ. Stud. 10(3):16–25. https://doi.org/10.5539/hes.v10n3p16

Al-Nasa’h M, Al-Tarawneh L, Awwad FMA, Ahmad I (2021) Estimating Students’ Online Learning Satisfaction during COVID-19: A Discriminant Analysis. Heliyon 7(12):1–7. https://doi.org/10.1016/j.helyon.2021.e08544

Al-Rahmi WM, Yahaya N, Aldraiweesh AA, Alamri MM, Aljarboa NA, Alturki U (2019) Integrating technology acceptance model with innovation diffusion theory: An empirical investigation on students’ intention to use E-learning systems. IEEE Access 7:26797–26809. https://doi.org/10.1109/ACCESS.2019.2899368

Aristovnik A, Keržič D, Ravšelj D, Tomaževič N, Umek L (2021) Impacts of the COVID-19 pandemic on life of higher education students: A global perspective. Sustainability 12(20):8438. https://doi.org/10.3390/su12208438

Article   CAS   Google Scholar  

Baber H (2021) Modelling the Acceptance of E-learning during the Pandemic Of COVID-19-A Study of South Korea. Int. J. Manag. Educ. 19(2):1–15. https://doi.org/10.1016/j.ijme.2021.100503

Article   MathSciNet   Google Scholar  

Bahasoan AN, Ayuandiani W, Mukhram M, Rahmat A (2020) Effectiveness of online learning in pandemic COVID-19. Int. J. Sci., Technol. Manag. 1(2):100–106

Google Scholar  

Balanskat A, Bingimlas KA (2006) Barriers to the Successful Integration of ICT in Teaching and Learning Environments: A Review of the Literature. Eurasia J. Math. Sci. Technol. Educ. 5(3):235–245. https://doi.org/10.12973/ejmste/75275

Bao W (2020) COVID-19 and Online Teaching in Higher Education: A Case Study of Peking University. Hum. Behav. Emerg. Technol. 2(2):113–115. https://doi.org/10.1002/hbe2.191

Article   PubMed   PubMed Central   Google Scholar  

Bernard RM et al. (2004) How Does Distance Education Compare with Classroom Instruction? A Meta-Analysis of the Empirical Literature. Rev. Educ. Res. 74(3):379–439

Boling EC, Hough M, Krinsky H, Saleem H, Stevens M (2012) Cutting the distance in distance education: Perspectives on what promotes positive, online learning experiences. Internet High. Educ. 15:118–126. https://doi.org/10.1016/j.iheduc.2011.11.006

Bozkurt A, Sharma RC (2020) Emergency remote teaching in a time of global crisis due to Coronavirus pandemic. Asian J. Distance Educ. 15(1):i–vi

Broadbent J (2017) Comparing online and blended learner’s self-regulated learning strategies and academic performance. Internet High. Educ. 33:24–32. https://doi.org/10.1016/j.iheduc.2017.01.004

Calaguas NP, Consunji PMP (2022) A Structural Equation Model Predicting Adults’ Online Learning Self-efficacy. Educ. Inf. Technol. 27:6233–6249. https://doi.org/10.1007/s10639-021-10871-y

Careaga-Butter M, Quintana MGB, Fuentes-Henríquez C (2020) Critical and Prospective Analysis of Online Education in Pandemic and Post-pandemic Contexts: Digital Tools and Resources to Support Teaching in synchronous and Asynchronous Learning Modalities. Aloma: evista de. psicologia, ciències de. l’educació i de. l’esport Blanquerna 38(2):23–32

Chakraborty M, Muyia Nafukho F (2014) Strengthening Student Engagement: What do Students Want in Online Courses? Eur. J. Train. Dev. 38(9):782–802

Chang V (2016) Review and Discussion: E-learning for Academia and Industry. Int. J. Inf. Manag. 36(3):476–485. https://doi.org/10.1016/j.ijinfomgt.2015.12.007

Chen YJ, Hsu LW (2022) Enhancing EFL Learners’ Self-efficacy Beliefs of Learning English with emoji Feedbacks in Call: Why and How. Behav. Sci. 12(7):227. https://doi.org/10.3390/bs12070227

Chuenyindee T, Montenegro LD, Ong AKS, Prasetyo YT, Nadlifatin R, Ayuwati ID, Sittiwatethanasiri T, Robas KPE (2022) The Perceived Usability of the Learning Management System during the COVID-19 Pandemic: Integrating System Usability Scale, Technology Acceptance Model, and Task-technology Fit. Work 73(1):41–58. https://doi.org/10.3233/WOR-220015

Article   PubMed   Google Scholar  

Crawford J, Butler-Henderson K, Rudolph J, Malkawi B, Glowatz M, Burton R, Lam S (2020) COVID-19: 20 countries’ higher education intra-period digital pedagogy responses. J. Appl. Teach. Learn. 3(1):120. https://doi.org/10.37074/jalt.2020.3.1.7

Dennen VP (2014) Becoming a blogger: Trajectories, norms, and activities in a community of practice. Computers Hum. Behav. 36:350–358. https://doi.org/10.1016/j.chb.2014.03.028

Dennen VP, Darabi AA, Smith LJ (2007) Instructor–learner Interaction in Online Courses: The Relative Perceived Importance of Particular Instructor Actions on Performance and Satisfaction. Distance Educ. 28(1):65–79

Dolan E, Hancock E, Wareing A (2015) An evaluation of online learning to teach practical competencies in undergraduate health science students. Internet High. Educ. 24:21–25

Garrison DR (2003) Cognitive Presence for Effective Asynchronous Online Learning: The Role of Reflective. Inq., Self-Direction Metacognition. Elem. Qual. Online Educ.: Pract. Direction 4(10):47–58

Gavranović V, Prodanović M (2021) ESP Teachers’ Perspectives on the Online Teaching Environment Imposed in the Covid-19 Era-A Case Study. N. Educ. Rev. 2:188–197. https://doi.org/10.15804/TNER.2021.64.2.15

Greenhow C, Lewin C, Staudt Willet KB (2021) The Educational Response to Covid-19 across Two Countries: a Critical Examination of Initial Digital Pedagogy Adoption. Technol., Pedagog. Educ. 30(1):7–25

Händel M, Stephan M, Gläser-Zikuda M, Kopp B, Bedenlier S, Ziegler A (2020) Digital readiness and its effects on higher education students’ socio-emotional perceptions in the context of the COVID-19 pandemic. J. Res. Technol. Educ. 53(2):1–13

Harper B (2018) Technology and Teacher-student Interactions: A Review of Empirical Research. J. Res. Technol. Educ. 50(3):214–225

Article   ADS   Google Scholar  

Hernández-Lara AB, Serradell-López E (2018) Student interactions in online discussion forums: their perception on learning with business simulation games. Behav. Inf. Technol. 37(4):419–429

Hodges C, Moore S, Lockee B, Trust T, Bond A (2020) The difference between emergency remote teaching and online learning. Educause Rev. 27:1–12

Hongsuchon T, Emary IMME, Hariguna T, Qhal EMA (2022) Assessing the Impact of Online-learning Effectiveness and Benefits in Knowledge Management, the Antecedent of Online-learning Strategies and Motivations: An Empirical Study. Sustainability 14(5):1–16. https://doi.org/10.3390/su14052570 . (2020)

Johnson N, Veletsianos G, Seaman J (2020) US faculty and administrators’ experiences and approaches in the early weeks of the COVID-19 pandemic. Online Learn. 24(2):6–21

Khojasteh L, Karimian Z, Farahmandi AY, Nasiri E, Salehi N (2023) E-content Development of English Language Courses during COVID-19: A Comprehensive Analysis of Students’ Satisfaction. J. Computer Educ. 10(1):107–133. https://doi.org/10.1007/s40692-022-00224-0

Kohnke, L, & Foung, D (2023). Exploring Microlearning for Teacher Professional Development: Voices from Hong Kong. In Tafazoli, D, M Picard (Eds.). Handbook of CALL Teacher Education Professional Development (pp. 279-292). Singapore: Springer Nature Singapore Pte Ltd

Lai CH, Lin HW, Lin RM, Tho PD (2019) Effect of Peer Interaction among Online Learning Community on Learning Engagement and Achievement. Int. J. Distance Educ. Technol. (IJDET) 17(1):66–77

Lashley PM, Sobers NP, Campbell MH, Emmanuel MK, Greaves N, Gittens-St Hilaire M, Murphy MM, Majumder MAA (2022) Student Satisfaction and Self-Efficacy in a Novel Online Clinical Clerkship Curriculum Delivered during the COVID-19 Pandemic. Adv. Med. Educ. Pract. 13:1029–1038. https://doi.org/10.2147/AMEP.S374133

Lee SJ, Srinivasan S, Trail T, Lewis D, Lopez S (2011) Examining the Relationship among Student Perception of Support, Course Satisfaction, and Learning Outcomes in Online Learning. Internet High. Educ. 14(3):158–163

Lowenthal PR, Dunlap JC (2020) Social Presence and Online Discussions: A Mixed Method Investigation. Distance Educ. 41:490–514. https://doi.org/10.1080/01587919.2020.1821603

Lucas M, Vicente PN (2023) A Double-edged Sword: Teachers’ Perceptions of the Benefits and Challenges of Online Learning and Learning in Higher Education. Educ. Inf. Technol. 23:5083–5103. https://doi.org/10.1007/s10639-022-11363-3

Maatuk AM, Elberkawi EK, Aljawarneh S, Rashaideh H, Alharbi H (2022) The COVID-19 Pandemic and E-learning: Challenges and Opportunities from the Perspective of Students and Instructors. J. Comput. High. Educ. 34:21–38. https://doi.org/10.1007/s12528-021-09274-2

Marinoni G, Van’t Land H, Jensen T (2020) The Impact of Covid-19 on Higher Education around the World. IAU Glob. Surv. Rep. 23:1–17

Martin F, Bolliger DU (2018) Engagement matters: Student perceptions on the importance of engagement strategies in the online learning environment. Online Learn. 22(1):205–222

Means B, et al. (2010). Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. Washington, D. C.: U.S. Department of Education

Miao J, Ma L (2022) Students’ Online Interaction, Self-regulation, and Learning Engagement in Higher Education: The Importance of Social Presence to Online Learning. Front. Psychol. 13:1–9. https://doi.org/10.3389/fpsyg.2022.815220

Mulla T, Munir S, Mohan V (2023) An Exploratory Study to Understand Faculty Members’ Perceptions and Challenges in Online Teaching. Int. Rev. Educ. 69:73–99. https://doi.org/10.1007/s11159-023-100

Ng EW (2018) Integrating Self-regulation Principles with Flipped Classroom Pedagogy for First Year University Students. Computer Educ. 126:65–74. https://doi.org/10.1007/s11409-011-9082-8

Ong SGT, Quek GCL (2023) Enhancing teacher–student interactions and student online engagement in an online learning environment. Learn. Environ. Res. 26:681–707. https://doi.org/10.1007/s10984-022-09447-5

Organisation for Economic Co‑operation and Development. (2015). The g20 skills strategy for developing and using skills for the 21st century. Retrieved from https://www.oecd.org/g20/topics/employment-andsocial-policy/The-G20-Skills-Strategy-for-Developing-and-Using-Skills-for-the-21st-Century.pdf

Paechter M, Maier B (2010) Online or face-to-face? Students’ experiences and preferences in e-learning. internet High. Educ. 13(4):292–297

Panigrahi R, Srivastava PR, Sharma D (2018) Online Learning: Adoption, Continuance, and Learning Outcome—A Review of Literature. Int. J. Inf. Manag. 43:1–14

Paul C, Diana M (2006) Teaching Methods and Time on Task in Junior Classrooms. Educ. Res. 30:90–97

Punie, Y, Zinnbauer, D, & Cabrera, M (2006). A review of the impact of ICT on learning. European Commission, Brussels, 6(5), 635-650

Radmer F, Goodchild S (2021) Online Mathematics Teaching and Learning during the COVID-19 Pandemic: The Perspective of Lecturers and Students. Nord. J. STEM Educ. 5(1):1–5. https://doi.org/10.5324/njsteme.v5i1.3914

Rapanta C, Botturi L, Goodyear P, Guadia L, Koole M (2021) Balancing Technology, Pedagogy and the New Normal: Post-pandemic Challenges for Higher Education. Postdigital Sci. Educ. 3:715–742. https://doi.org/10.1007/s42438-021-00249-1

Rasheed, RA, Kamsin, A, & Abdullah, NA (2020) Challenges in the Online Component of Blended Learning: A Systematic Review. Computers & Education, 144. https://doi.org/10.1016/j.compedu.2019.103701

Schaufeli W, Salanova M, Gonzalez-Roma V (2002) The Measurement of Engagement and Burnout: A Two Sample Confirmatory Factor Analytic Approach. J. Happiness Stud. 3:71–92

Song L, Singleton ES, Hill JR, Koh MH (2004) Improving online learning: Student perceptions of useful and challenging characteristics. Internet High. Educ. 7(1):59–70. https://doi.org/10.1016/j.iheduc.2003.11.003

Tan HT, Chan PP, Said NM (2021) Higher Education Students’ Online Instruction Perceptions: A Quality Virtual Learning Environment. Sustainability 13:10840. https://doi.org/10.3390/su131910840

Tenório T, Bittencourt II, Isotani S, Silva AP (2016) Does peer assessment in online learning environments work? A systematic review of the literature. Computers Hum. Behav. 64:94–107. https://doi.org/10.1016/j.chb.2016.06.020

Torsani, S (2023) Teacher Education in Mobile Assisted Language Learning for Adult Migrants: A Study of Provincial Centers for Adult Education in Italy. In Tafazoli, D, & M Picard (eds.). Handbook of CALL Teacher Education Professional Development (pp. 179-192). Singapore: Springer Nature Singapore Pte Ltd

Trust T, Whalen J (2020) Should teachers be trained in emergency remote teaching? Lessons learned from the COVID-19 pandemic. J. Technol. Teach. Educ. 28(2):189–199

UNESCO (2020a) COVID-19 Impact on education. UNESCO. Retrieved from https://en.unesco.org/covid19/educationresponse

UNESCO (2020b) Education: From Disruption to Recovery. UNESCO. Retrieved from https://en.unesco.org/covid19/educationresponse

Wang M, & Wang F (2021, August) Comparative Analysis of University Education Effect under the Traditional Teaching and Online Teaching Mode. In The Sixth International Conference on Information Management and Technology (pp. 1-6)

Xu Q, Wu J, Peng HY (2022) Chinese EFL University Students’ Self-efficacy for Online Self-regulated Learning: Dynamic Features and Influencing Factors. Front. Psychol. 13:1–12. https://doi.org/10.3389/fpsyg.2022.912970

Zheng RK, Li F, Jiang L, Li SM (2022) Analysis of the Current Situation and Driving Factors of College Students’ Autonomous Learning in the Network Environment. Front. Humanit. Soc. Sci. 2(7):44–50

Zhu XM, Gong Q, Wang Q, He YJ, Sun ZQ, Liu FF (2023) Analysis of Students’ Online Learning Engagement during the COVID-19 Pandemic: A Case Study of a SPOC-Based Geography Education Undergraduate Course. Sustainability 15(5):4544. https://doi.org/10.3390/su15054544

Zhu Y, Geng G, Disney L, Pan Zihao (2023) Changes in University Students’ Behavioral intention to learn online throughout the COVID-19: Insights for Online Teaching in the Post-pandemic Era. Educ. Inf. Technol. 28:3859–3892. https://doi.org/10.1007/s10639-022-11320-0

Zhu YH, Xu YY, Wang XY, Yan SY, Zhao L (2022) The Selectivity and Suitability of Online Learning Resources as Predictor of the Effects of Self-efficacy on Teacher Satisfaction During the COVID-19 Lockdown. Front. Psychol. 13:1–11. https://doi.org/10.3389/fpsyg.2022.765832

Zimmerman WA, Kulikowich JM (2016) Online Learning Self-efficacy in Students with and Without Online Learning Experience. Am. J. Distance Educ. 30(3):180–190. https://doi.org/10.1080/08923647.2016.1193801

Download references

Acknowledgements

The Corresponding author received the National Social Science Foundation of China for Education General Program (BGA210054) for this work.

Author information

Authors and affiliations.

School of Foreign Studies, Wenzhou University, Wenzhou, China

Yujie Su, Xiaoshu Xu & Xinyu Xu

Faculty of Languages and Translation, Macao Polytechnic University, Macao, China

Yunfeng Zhang

Faculty of Applied Sciences, Macao Polytechnic University, Macao, China

Shanshan Hao

You can also search for this author in PubMed   Google Scholar

Contributions

XSX was responsible for conceptualization and, alongside YFZ, for data curation. YJS and XYX conducted the formal analysis. Funding acquisition was managed by YFZ. The investigation was carried out by YJS and YFZ. Methodology development was a collaboration between YJS and XSX. XSX and YJS also managed project administration, with additional resource management by SSH and XYX. YJS handled the software aspect, and supervision was overseen by XSX. SSH and XYX were responsible for validation, and visualization was managed by YJS. The original draft was written by XSX and YJS, while the review and editing were conducted by YFZ and SSH.

Corresponding author

Correspondence to Xiaoshu Xu .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

The questionnaire and methodology for this study were approved by the Human Research Ethics Committee of Wenzhou University (Ethics approval number WGY202302).

Informed consent

Informed consent was diligently obtained from all participants involved in the study, ensuring adherence to ethical standards. Detailed consent forms, outlining the study’s scope and participants’ rights, were signed by participants. Documentation of this process is well-maintained and can be provided upon request during peer review or post-publication.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Su, Y., Xu, X., Zhang, Y. et al. Looking back to move forward: comparison of instructors’ and undergraduates’ retrospection on the effectiveness of online learning using the nine-outcome influencing factors. Humanit Soc Sci Commun 11 , 594 (2024). https://doi.org/10.1057/s41599-024-03097-z

Download citation

Received : 24 October 2023

Accepted : 17 April 2024

Published : 09 May 2024

DOI : https://doi.org/10.1057/s41599-024-03097-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Elsevier - PMC COVID-19 Collection

A systematic review of research on online teaching and learning from 2009 to 2018

Associated data.

Systematic reviews were conducted in the nineties and early 2000's on online learning research. However, there is no review examining the broader aspect of research themes in online learning in the last decade. This systematic review addresses this gap by examining 619 research articles on online learning published in twelve journals in the last decade. These studies were examined for publication trends and patterns, research themes, research methods, and research settings and compared with the research themes from the previous decades. While there has been a slight decrease in the number of studies on online learning in 2015 and 2016, it has then continued to increase in 2017 and 2018. The majority of the studies were quantitative in nature and were examined in higher education. Online learning research was categorized into twelve themes and a framework across learner, course and instructor, and organizational levels was developed. Online learner characteristics and online engagement were examined in a high number of studies and were consistent with three of the prior systematic reviews. However, there is still a need for more research on organization level topics such as leadership, policy, and management and access, culture, equity, inclusion, and ethics and also on online instructor characteristics.

• • Twelve online learning research themes were identified in 2009–2018.
• • A framework with learner, course and instructor, and organizational levels was used.
• • Online learner characteristics and engagement were the mostly examined themes.
• • The majority of the studies used quantitative research methods and in higher education.
• • There is a need for more research on organization level topics.

1. Introduction

Online learning has been on the increase in the last two decades. In the United States, though higher education enrollment has declined, online learning enrollment in public institutions has continued to increase ( Allen & Seaman, 2017 ), and so has the research on online learning. There have been review studies conducted on specific areas on online learning such as innovations in online learning strategies ( Davis et al., 2018 ), empirical MOOC literature ( Liyanagunawardena et al., 2013 ; Veletsianos & Shepherdson, 2016 ; Zhu et al., 2018 ), quality in online education ( Esfijani, 2018 ), accessibility in online higher education ( Lee, 2017 ), synchronous online learning ( Martin et al., 2017 ), K-12 preparation for online teaching ( Moore-Adams et al., 2016 ), polychronicity in online learning ( Capdeferro et al., 2014 ), meaningful learning research in elearning and online learning environments ( Tsai, Shen, & Chiang, 2013 ), problem-based learning in elearning and online learning environments ( Tsai & Chiang, 2013 ), asynchronous online discussions ( Thomas, 2013 ), self-regulated learning in online learning environments ( Tsai, Shen, & Fan, 2013 ), game-based learning in online learning environments ( Tsai & Fan, 2013 ), and online course dropout ( Lee & Choi, 2011 ). While there have been review studies conducted on specific online learning topics, very few studies have been conducted on the broader aspect of online learning examining research themes.

2. Systematic Reviews of Distance Education and Online Learning Research

Distance education has evolved from offline to online settings with the access to internet and COVID-19 has made online learning the common delivery method across the world. Tallent-Runnels et al. (2006) reviewed research late 1990's to early 2000's, Berge and Mrozowski (2001) reviewed research 1990 to 1999, and Zawacki-Richter et al. (2009) reviewed research in 2000–2008 on distance education and online learning. Table 1 shows the research themes from previous systematic reviews on online learning research. There are some themes that re-occur in the various reviews, and there are also new themes that emerge. Though there have been reviews conducted in the nineties and early 2000's, there is no review examining the broader aspect of research themes in online learning in the last decade. Hence, the need for this systematic review which informs the research themes in online learning from 2009 to 2018. In the following sections, we review these systematic review studies in detail.

Comparison of online learning research themes from previous studies.

2.1. Distance education research themes, 1990 to 1999 ( Berge & Mrozowski, 2001 )

Berge and Mrozowski (2001) reviewed 890 research articles and dissertation abstracts on distance education from 1990 to 1999. The four distance education journals chosen by the authors to represent distance education included, American Journal of Distance Education, Distance Education, Open Learning, and the Journal of Distance Education. This review overlapped in the dates of the Tallent-Runnels et al. (2006) study. Berge and Mrozowski (2001) categorized the articles according to Sherry's (1996) ten themes of research issues in distance education: redefining roles of instructor and students, technologies used, issues of design, strategies to stimulate learning, learner characteristics and support, issues related to operating and policies and administration, access and equity, and costs and benefits.

In the Berge and Mrozowski (2001) study, more than 100 studies focused on each of the three themes: (1) design issues, (2) learner characteristics, and (3) strategies to increase interactivity and active learning. By design issues, the authors focused on instructional systems design and focused on topics such as content requirement, technical constraints, interactivity, and feedback. The next theme, strategies to increase interactivity and active learning, were closely related to design issues and focused on students’ modes of learning. Learner characteristics focused on accommodating various learning styles through customized instructional theory. Less than 50 studies focused on the three least examined themes: (1) cost-benefit tradeoffs, (2) equity and accessibility, and (3) learner support. Cost-benefit trade-offs focused on the implementation costs of distance education based on school characteristics. Equity and accessibility focused on the equity of access to distance education systems. Learner support included topics such as teacher to teacher support as well as teacher to student support.

2.2. Online learning research themes, 1993 to 2004 ( Tallent-Runnels et al., 2006 )

Tallent-Runnels et al. (2006) reviewed research on online instruction from 1993 to 2004. They reviewed 76 articles focused on online learning by searching five databases, ERIC, PsycINFO, ContentFirst, Education Abstracts, and WilsonSelect. Tallent-Runnels et al. (2006) categorized research into four themes, (1) course environment, (2) learners' outcomes, (3) learners’ characteristics, and (4) institutional and administrative factors. The first theme that the authors describe as course environment ( n  = 41, 53.9%) is an overarching theme that includes classroom culture, structural assistance, success factors, online interaction, and evaluation.

Tallent-Runnels et al. (2006) for their second theme found that studies focused on questions involving the process of teaching and learning and methods to explore cognitive and affective learner outcomes ( n  = 29, 38.2%). The authors stated that they found the research designs flawed and lacked rigor. However, the literature comparing traditional and online classrooms found both delivery systems to be adequate. Another research theme focused on learners’ characteristics ( n  = 12, 15.8%) and the synergy of learners, design of the online course, and system of delivery. Research findings revealed that online learners were mainly non-traditional, Caucasian, had different learning styles, and were highly motivated to learn. The final theme that they reported was institutional and administrative factors (n  = 13, 17.1%) on online learning. Their findings revealed that there was a lack of scholarly research in this area and most institutions did not have formal policies in place for course development as well as faculty and student support in training and evaluation. Their research confirmed that when universities offered online courses, it improved student enrollment numbers.

2.3. Distance education research themes 2000 to 2008 ( Zawacki-Richter et al., 2009 )

Zawacki-Richter et al. (2009) reviewed 695 articles on distance education from 2000 to 2008 using the Delphi method for consensus in identifying areas and classified the literature from five prominent journals. The five journals selected due to their wide scope in research in distance education included Open Learning, Distance Education, American Journal of Distance Education, the Journal of Distance Education, and the International Review of Research in Open and Distributed Learning. The reviewers examined the main focus of research and identified gaps in distance education research in this review.

Zawacki-Richter et al. (2009) classified the studies into macro, meso and micro levels focusing on 15 areas of research. The five areas of the macro-level addressed: (1) access, equity and ethics to deliver distance education for developing nations and the role of various technologies to narrow the digital divide, (2) teaching and learning drivers, markets, and professional development in the global context, (3) distance delivery systems and institutional partnerships and programs and impact of hybrid modes of delivery, (4) theoretical frameworks and models for instruction, knowledge building, and learner interactions in distance education practice, and (5) the types of preferred research methodologies. The meso-level focused on seven areas that involve: (1) management and organization for sustaining distance education programs, (2) examining financial aspects of developing and implementing online programs, (3) the challenges and benefits of new technologies for teaching and learning, (4) incentives to innovate, (5) professional development and support for faculty, (6) learner support services, and (7) issues involving quality standards and the impact on student enrollment and retention. The micro-level focused on three areas: (1) instructional design and pedagogical approaches, (2) culturally appropriate materials, interaction, communication, and collaboration among a community of learners, and (3) focus on characteristics of adult learners, socio-economic backgrounds, learning preferences, and dispositions.

The top three research themes in this review by Zawacki-Richter et al. (2009) were interaction and communities of learning ( n  = 122, 17.6%), instructional design ( n  = 121, 17.4%) and learner characteristics ( n  = 113, 16.3%). The lowest number of studies (less than 3%) were found in studies examining the following research themes, management and organization ( n  = 18), research methods in DE and knowledge transfer ( n  = 13), globalization of education and cross-cultural aspects ( n  = 13), innovation and change ( n  = 13), and costs and benefits ( n  = 12).

2.4. Online learning research themes

These three systematic reviews provide a broad understanding of distance education and online learning research themes from 1990 to 2008. However, there is an increase in the number of research studies on online learning in this decade and there is a need to identify recent research themes examined. Based on the previous systematic reviews ( Berge & Mrozowski, 2001 ; Hung, 2012 ; Tallent-Runnels et al., 2006 ; Zawacki-Richter et al., 2009 ), online learning research in this study is grouped into twelve different research themes which include Learner characteristics, Instructor characteristics, Course or program design and development, Course Facilitation, Engagement, Course Assessment, Course Technologies, Access, Culture, Equity, Inclusion, and Ethics, Leadership, Policy and Management, Instructor and Learner Support, and Learner Outcomes. Table 2 below describes each of the research themes and using these themes, a framework is derived in Fig. 1 .

Research themes in online learning.

Online learning research themes framework.

The collection of research themes is presented as a framework in Fig. 1 . The themes are organized by domain or level to underscore the nested relationship that exists. As evidenced by the assortment of themes, research can focus on any domain of delivery or associated context. The “Learner” domain captures characteristics and outcomes related to learners and their interaction within the courses. The “Course and Instructor” domain captures elements about the broader design of the course and facilitation by the instructor, and the “Organizational” domain acknowledges the contextual influences on the course. It is important to note as well that due to the nesting, research themes can cross domains. For example, the broader cultural context may be studied as it pertains to course design and development, and institutional support can include both learner support and instructor support. Likewise, engagement research can involve instructors as well as learners.

In this introduction section, we have reviewed three systematic reviews on online learning research ( Berge & Mrozowski, 2001 ; Tallent-Runnels et al., 2006 ; Zawacki-Richter et al., 2009 ). Based on these reviews and other research, we have derived twelve themes to develop an online learning research framework which is nested in three levels: learner, course and instructor, and organization.

2.5. Purpose of this research

In two out of the three previous reviews, design, learner characteristics and interaction were examined in the highest number of studies. On the other hand, cost-benefit tradeoffs, equity and accessibility, institutional and administrative factors, and globalization and cross-cultural aspects were examined in the least number of studies. One explanation for this may be that it is a function of nesting, noting that studies falling in the Organizational and Course levels may encompass several courses or many more participants within courses. However, while some research themes re-occur, there are also variations in some themes across time, suggesting the importance of research themes rise and fall over time. Thus, a critical examination of the trends in themes is helpful for understanding where research is needed most. Also, since there is no recent study examining online learning research themes in the last decade, this study strives to address that gap by focusing on recent research themes found in the literature, and also reviewing research methods and settings. Notably, one goal is to also compare findings from this decade to the previous review studies. Overall, the purpose of this study is to examine publication trends in online learning research taking place during the last ten years and compare it with the previous themes identified in other review studies. Due to the continued growth of online learning research into new contexts and among new researchers, we also examine the research methods and settings found in the studies of this review.

The following research questions are addressed in this study.

• 1. What percentage of the population of articles published in the journals reviewed from 2009 to 2018 were related to online learning and empirical?
• 2. What is the frequency of online learning research themes in the empirical online learning articles of journals reviewed from 2009 to 2018?
• 3. What is the frequency of research methods and settings that researchers employed in the empirical online learning articles of the journals reviewed from 2009 to 2018?

This five-step systematic review process described in the U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse Procedures and Standards Handbook, Version 4.0 ( 2017 ) was used in this systematic review: (a) developing the review protocol, (b) identifying relevant literature, (c) screening studies, (d) reviewing articles, and (e) reporting findings.

3.1. Data sources and search strategies

The Education Research Complete database was searched using the keywords below for published articles between the years 2009 and 2018 using both the Title and Keyword function for the following search terms.

“online learning" OR "online teaching" OR "online program" OR "online course" OR “online education”

3.2. Inclusion/exclusion criteria

The initial search of online learning research among journals in the database resulted in more than 3000 possible articles. Therefore, we limited our search to select journals that focus on publishing peer-reviewed online learning and educational research. Our aim was to capture the journals that published the most articles in online learning. However, we also wanted to incorporate the concept of rigor, so we used expert perception to identify 12 peer-reviewed journals that publish high-quality online learning research. Dissertations and conference proceedings were excluded. To be included in this systematic review, each study had to meet the screening criteria as described in Table 3 . A research study was excluded if it did not meet all of the criteria to be included.

Inclusion/Exclusion criteria.

3.3. Process flow selection of articles

Fig. 2 shows the process flow involved in the selection of articles. The search in the database Education Research Complete yielded an initial sample of 3332 articles. Targeting the 12 journals removed 2579 articles. After reviewing the abstracts, we removed 134 articles based on the inclusion/exclusion criteria. The final sample, consisting of 619 articles, was entered into the computer software MAXQDA ( VERBI Software, 2019 ) for coding.

Flowchart of online learning research selection.

3.4. Developing review protocol

A review protocol was designed as a codebook in MAXQDA ( VERBI Software, 2019 ) by the three researchers. The codebook was developed based on findings from the previous review studies and from the initial screening of the articles in this review. The codebook included 12 research themes listed earlier in Table 2 (Learner characteristics, Instructor characteristics, Course or program design and development, Course Facilitation, Engagement, Course Assessment, Course Technologies, Access, Culture, Equity, Inclusion, and Ethics, Leadership, Policy and Management, Instructor and Learner Support, and Learner Outcomes), four research settings (higher education, continuing education, K-12, corporate/military), and three research designs (quantitative, qualitative and mixed methods). Fig. 3 below is a screenshot of MAXQDA used for the coding process.

Codebook from MAXQDA.

3.5. Data coding

Research articles were coded by two researchers in MAXQDA. Two researchers independently coded 10% of the articles and then discussed and updated the coding framework. The second author who was a doctoral student coded the remaining studies. The researchers met bi-weekly to address coding questions that emerged. After the first phase of coding, we found that more than 100 studies fell into each of the categories of Learner Characteristics or Engagement, so we decided to pursue a second phase of coding and reexamine the two themes. Learner Characteristics were classified into the subthemes of Academic, Affective, Motivational, Self-regulation, Cognitive, and Demographic Characteristics. Engagement was classified into the subthemes of Collaborating, Communication, Community, Involvement, Interaction, Participation, and Presence.

3.6. Data analysis

Frequency tables were generated for each of the variables so that outliers could be examined and narrative data could be collapsed into categories. Once cleaned and collapsed into a reasonable number of categories, descriptive statistics were used to describe each of the coded elements. We first present the frequencies of publications related to online learning in the 12 journals. The total number of articles for each journal (collectively, the population) was hand-counted from journal websites, excluding editorials and book reviews. The publication trend of online learning research was also depicted from 2009 to 2018. Then, the descriptive information of the 12 themes, including the subthemes of Learner Characteristics and Engagement were provided. Finally, research themes by research settings and methodology were elaborated.

4.1. Publication trends on online learning

Publication patterns of the 619 articles reviewed from the 12 journals are presented in Table 4 . International Review of Research in Open and Distributed Learning had the highest number of publications in this review. Overall, about 8% of the articles appearing in these twelve journals consisted of online learning publications; however, several journals had concentrations of online learning articles totaling more than 20%.

Empirical online learning research articles by journal, 2009–2018.

Note . Journal's Total Article count excludes reviews and editorials.

The publication trend of online learning research is depicted in Fig. 4 . When disaggregated by year, the total frequency of publications shows an increasing trend. Online learning articles increased throughout the decade and hit a relative maximum in 2014. The greatest number of online learning articles ( n  = 86) occurred most recently, in 2018.

Online learning publication trends by year.

4.2. Online learning research themes that appeared in the selected articles

The publications were categorized into the twelve research themes identified in Fig. 1 . The frequency counts and percentages of the research themes are provided in Table 5 below. A majority of the research is categorized into the Learner domain. The fewest number of articles appears in the Organization domain.

Research themes in the online learning publications from 2009 to 2018.

The specific themes of Engagement ( n  = 179, 28.92%) and Learner Characteristics ( n  = 134, 21.65%) were most often examined in publications. These two themes were further coded to identify sub-themes, which are described in the next two sections. Publications focusing on Instructor Characteristics ( n  = 21, 3.39%) were least common in the dataset.

4.2.1. Research on engagement

The largest number of studies was on engagement in online learning, which in the online learning literature is referred to and examined through different terms. Hence, we explore this category in more detail. In this review, we categorized the articles into seven different sub-themes as examined through different lenses including presence, interaction, community, participation, collaboration, involvement, and communication. We use the term “involvement” as one of the terms since researchers sometimes broadly used the term engagement to describe their work without further description. Table 6 below provides the description, frequency, and percentages of the various studies related to engagement.

Research sub-themes on engagement.

In the sections below, we provide several examples of the different engagement sub-themes that were studied within the larger engagement theme.

Presence. This sub-theme was the most researched in engagement. With the development of the community of inquiry framework most of the studies in this subtheme examined social presence ( Akcaoglu & Lee, 2016 ; Phirangee & Malec, 2017 ; Wei et al., 2012 ), teaching presence ( Orcutt & Dringus, 2017 ; Preisman, 2014 ; Wisneski et al., 2015 ) and cognitive presence ( Archibald, 2010 ; Olesova et al., 2016 ).

Interaction . This was the second most studied theme under engagement. Researchers examined increasing interpersonal interactions ( Cung et al., 2018 ), learner-learner interactions ( Phirangee, 2016 ; Shackelford & Maxwell, 2012 ; Tawfik et al., 2018 ), peer-peer interaction ( Comer et al., 2014 ), learner-instructor interaction ( Kuo et al., 2014 ), learner-content interaction ( Zimmerman, 2012 ), interaction through peer mentoring ( Ruane & Koku, 2014 ), interaction and community building ( Thormann & Fidalgo, 2014 ), and interaction in discussions ( Ruane & Lee, 2016 ; Tibi, 2018 ).

Community. Researchers examined building community in online courses ( Berry, 2017 ), supporting a sense of community ( Jiang, 2017 ), building an online learning community of practice ( Cho, 2016 ), building an academic community ( Glazer & Wanstreet, 2011 ; Nye, 2015 ; Overbaugh & Nickel, 2011 ), and examining connectedness and rapport in an online community ( Bolliger & Inan, 2012 ; Murphy & Rodríguez-Manzanares, 2012 ; Slagter van Tryon & Bishop, 2012 ).

Participation. Researchers examined engagement through participation in a number of studies. Some of the topics include, participation patterns in online discussion ( Marbouti & Wise, 2016 ; Wise et al., 2012 ), participation in MOOCs ( Ahn et al., 2013 ; Saadatmand & Kumpulainen, 2014 ), features that influence students’ online participation ( Rye & Støkken, 2012 ) and active participation.

Collaboration. Researchers examined engagement through collaborative learning. Specific studies focused on cross-cultural collaboration ( Kumi-Yeboah, 2018 ; Yang et al., 2014 ), how virtual teams collaborate ( Verstegen et al., 2018 ), types of collaboration teams ( Wicks et al., 2015 ), tools for collaboration ( Boling et al., 2014 ), and support for collaboration ( Kopp et al., 2012 ).

Involvement. Researchers examined engaging learners through involvement in various learning activities ( Cundell & Sheepy, 2018 ), student engagement through various measures ( Dixson, 2015 ), how instructors included engagement to involve students in learning ( O'Shea et al., 2015 ), different strategies to engage the learner ( Amador & Mederer, 2013 ), and designed emotionally engaging online environments ( Koseoglu & Doering, 2011 ).

Communication. Researchers examined communication in online learning in studies using social network analysis ( Ergün & Usluel, 2016 ), using informal communication tools such as Facebook for class discussion ( Kent, 2013 ), and using various modes of communication ( Cunningham et al., 2010 ; Rowe, 2016 ). Studies have also focused on both asynchronous and synchronous aspects of communication ( Swaggerty & Broemmel, 2017 ; Yamagata-Lynch, 2014 ).

4.2.2. Research on learner characteristics

The second largest theme was learner characteristics. In this review, we explore this further to identify several aspects of learner characteristics. In this review, we categorized the learner characteristics into self-regulation characteristics, motivational characteristics, academic characteristics, affective characteristics, cognitive characteristics, and demographic characteristics. Table 7 provides the number of studies and percentages examining the various learner characteristics.

Research sub-themes on learner characteristics.

Online learning has elements that are different from the traditional face-to-face classroom and so the characteristics of the online learners are also different. Yukselturk and Top (2013) categorized online learner profile into ten aspects: gender, age, work status, self-efficacy, online readiness, self-regulation, participation in discussion list, participation in chat sessions, satisfaction, and achievement. Their categorization shows that there are differences in online learner characteristics in these aspects when compared to learners in other settings. Some of the other aspects such as participation and achievement as discussed by Yukselturk and Top (2013) are discussed in different research themes in this study. The sections below provide examples of the learner characteristics sub-themes that were studied.

Self-regulation. Several researchers have examined self-regulation in online learning. They found that successful online learners are academically motivated ( Artino & Stephens, 2009 ), have academic self-efficacy ( Cho & Shen, 2013 ), have grit and intention to succeed ( Wang & Baker, 2018 ), have time management and elaboration strategies ( Broadbent, 2017 ), set goals and revisit course content ( Kizilcec et al., 2017 ), and persist ( Glazer & Murphy, 2015 ). Researchers found a positive relationship between learner's self-regulation and interaction ( Delen et al., 2014 ) and self-regulation and communication and collaboration ( Barnard et al., 2009 ).

Motivation. Researchers focused on motivation of online learners including different motivation levels of online learners ( Li & Tsai, 2017 ), what motivated online learners ( Chaiprasurt & Esichaikul, 2013 ), differences in motivation of online learners ( Hartnett et al., 2011 ), and motivation when compared to face to face learners ( Paechter & Maier, 2010 ). Harnett et al. (2011) found that online learner motivation was complex, multifaceted, and sensitive to situational conditions.

Academic. Several researchers have focused on academic aspects for online learner characteristics. Readiness for online learning has been examined as an academic factor by several researchers ( Buzdar et al., 2016 ; Dray et al., 2011 ; Wladis & Samuels, 2016 ; Yu, 2018 ) specifically focusing on creating and validating measures to examine online learner readiness including examining students emotional intelligence as a measure of student readiness for online learning. Researchers have also examined other academic factors such as academic standing ( Bradford & Wyatt, 2010 ), course level factors ( Wladis et al., 2014 ) and academic skills in online courses ( Shea & Bidjerano, 2014 ).

Affective. Anderson and Bourke (2013) describe affective characteristics through which learners express feelings or emotions. Several research studies focused on the affective characteristics of online learners. Learner satisfaction for online learning has been examined by several researchers ( Cole et al., 2014 ; Dziuban et al., 2015 ; Kuo et al., 2013 ; Lee, 2014a ) along with examining student emotions towards online assessment ( Kim et al., 2014 ).

Cognitive. Researchers have also examined cognitive aspects of learner characteristics including meta-cognitive skills, cognitive variables, higher-order thinking, cognitive density, and critical thinking ( Chen & Wu, 2012 ; Lee, 2014b ). Lee (2014b) examined the relationship between cognitive presence density and higher-order thinking skills. Chen and Wu (2012) examined the relationship between cognitive and motivational variables in an online system for secondary physical education.

Demographic. Researchers have examined various demographic factors in online learning. Several researchers have examined gender differences in online learning ( Bayeck et al., 2018 ; Lowes et al., 2016 ; Yukselturk & Bulut, 2009 ), ethnicity, age ( Ke & Kwak, 2013 ), and minority status ( Yeboah & Smith, 2016 ) of online learners.

4.2.3. Less frequently studied research themes

While engagement and learner characteristics were studied the most, other themes were less often studied in the literature and are presented here, according to size, with general descriptions of the types of research examined for each.

Evaluation and Quality Assurance. There were 38 studies (6.14%) published in the theme of evaluation and quality assurance. Some of the studies in this theme focused on course quality standards, using quality matters to evaluate quality, using the CIPP model for evaluation, online learning system evaluation, and course and program evaluations.

Course Technologies. There were 35 studies (5.65%) published in the course technologies theme. Some of the studies examined specific technologies such as Edmodo, YouTube, Web 2.0 tools, wikis, Twitter, WebCT, Screencasts, and Web conferencing systems in the online learning context.

Course Facilitation. There were 34 studies (5.49%) published in the course facilitation theme. Some of the studies in this theme examined facilitation strategies and methods, experiences of online facilitators, and online teaching methods.

Institutional Support. There were 33 studies (5.33%) published in the institutional support theme which included support for both the instructor and learner. Some of the studies on instructor support focused on training new online instructors, mentoring programs for faculty, professional development resources for faculty, online adjunct faculty training, and institutional support for online instructors. Studies on learner support focused on learning resources for online students, cognitive and social support for online learners, and help systems for online learner support.

Learner Outcome. There were 32 studies (5.17%) published in the learner outcome theme. Some of the studies that were examined in this theme focused on online learner enrollment, completion, learner dropout, retention, and learner success.

Course Assessment. There were 30 studies (4.85%) published in the course assessment theme. Some of the studies in the course assessment theme examined online exams, peer assessment and peer feedback, proctoring in online exams, and alternative assessments such as eportfolio.

Access, Culture, Equity, Inclusion, and Ethics. There were 29 studies (4.68%) published in the access, culture, equity, inclusion, and ethics theme. Some of the studies in this theme examined online learning across cultures, multi-cultural effectiveness, multi-access, and cultural diversity in online learning.

Leadership, Policy, and Management. There were 27 studies (4.36%) published in the leadership, policy, and management theme. Some of the studies on leadership, policy, and management focused on online learning leaders, stakeholders, strategies for online learning leadership, resource requirements, university policies for online course policies, governance, course ownership, and faculty incentives for online teaching.

Course Design and Development. There were 27 studies (4.36%) published in the course design and development theme. Some of the studies examined in this theme focused on design elements, design issues, design process, design competencies, design considerations, and instructional design in online courses.

Instructor Characteristics. There were 21 studies (3.39%) published in the instructor characteristics theme. Some of the studies in this theme were on motivation and experiences of online instructors, ability to perform online teaching duties, roles of online instructors, and adjunct versus full-time online instructors.

4.3. Research settings and methodology used in the studies

The research methods used in the studies were classified into quantitative, qualitative, and mixed methods ( Harwell, 2012 , pp. 147–163). The research setting was categorized into higher education, continuing education, K-12, and corporate/military. As shown in Table A in the appendix, the vast majority of the publications used higher education as the research setting ( n  = 509, 67.6%). Table B in the appendix shows that approximately half of the studies adopted the quantitative method ( n  = 324, 43.03%), followed by the qualitative method ( n  = 200, 26.56%). Mixed methods account for the smallest portion ( n  = 95, 12.62%).

Table A shows that the patterns of the four research settings were approximately consistent across the 12 themes except for the theme of Leaner Outcome and Institutional Support. Continuing education had a higher relative frequency in Learner Outcome (0.28) and K-12 had a higher relative frequency in Institutional Support (0.33) compared to the frequencies they had in the total themes (0.09 and 0.08 respectively). Table B in the appendix shows that the distribution of the three methods were not consistent across the 12 themes. While quantitative studies and qualitative studies were roughly evenly distributed in Engagement, they had a large discrepancy in Learner Characteristics. There were 100 quantitative studies; however, only 18 qualitative studies published in the theme of Learner Characteristics.

In summary, around 8% of the articles published in the 12 journals focus on online learning. Online learning publications showed a tendency of increase on the whole in the past decade, albeit fluctuated, with the greatest number occurring in 2018. Among the 12 research themes related to online learning, the themes of Engagement and Learner Characteristics were studied the most and the theme of Instructor Characteristics was studied the least. Most studies were conducted in the higher education setting and approximately half of the studies used the quantitative method. Looking at the 12 themes by setting and method, we found that the patterns of the themes by setting or by method were not consistent across the 12 themes.

The quality of our findings was ensured by scientific and thorough searches and coding consistency. The selection of the 12 journals provides evidence of the representativeness and quality of primary studies. In the coding process, any difficulties and questions were resolved by consultations with the research team at bi-weekly meetings, which ensures the intra-rater and interrater reliability of coding. All these approaches guarantee the transparency and replicability of the process and the quality of our results.

5. Discussion

This review enabled us to identify the online learning research themes examined from 2009 to 2018. In the section below, we review the most studied research themes, engagement and learner characteristics along with implications, limitations, and directions for future research.

5.1. Most studied research themes

Three out of the four systematic reviews informing the design of the present study found that online learner characteristics and online engagement were examined in a high number of studies. In this review, about half of the studies reviewed (50.57%) focused on online learner characteristics or online engagement. This shows the continued importance of these two themes. In the Tallent-Runnels et al.’s (2006) study, the learner characteristics theme was identified as least studied for which they state that researchers are beginning to investigate learner characteristics in the early days of online learning.

One of the differences found in this review is that course design and development was examined in the least number of studies in this review compared to two prior systematic reviews ( Berge & Mrozowski, 2001 ; Zawacki-Richter et al., 2009 ). Zawacki-Richter et al. did not use a keyword search but reviewed all the articles in five different distance education journals. Berge and Mrozowski (2001) included a research theme called design issues to include all aspects of instructional systems design in distance education journals. In our study, in addition to course design and development, we also had focused themes on learner outcomes, course facilitation, course assessment and course evaluation. These are all instructional design focused topics and since we had multiple themes focusing on instructional design topics, the course design and development category might have resulted in fewer studies. There is still a need for more studies to focus on online course design and development.

5.2. Least frequently studied research themes

Three out of the four systematic reviews discussed in the opening of this study found management and organization factors to be least studied. In this review, Leadership, Policy, and Management was studied among 4.36% of the studies and Access, Culture, Equity, Inclusion, and Ethics was studied among 4.68% of the studies in the organizational level. The theme on Equity and accessibility was also found to be the least studied theme in the Berge and Mrozowski (2001) study. In addition, instructor characteristics was the least examined research theme among the twelve themes studied in this review. Only 3.39% of the studies were on instructor characteristics. While there were some studies examining instructor motivation and experiences, instructor ability to teach online, online instructor roles, and adjunct versus full-time online instructors, there is still a need to examine topics focused on instructors and online teaching. This theme was not included in the prior reviews as the focus was more on the learner and the course but not on the instructor. While it is helpful to see research evolving on instructor focused topics, there is still a need for more research on the online instructor.

5.3. Comparing research themes from current study to previous studies

The research themes from this review were compared with research themes from previous systematic reviews, which targeted prior decades. Table 8 shows the comparison.

Comparison of most and least studied online learning research themes from current to previous reviews.

L = Learner, C=Course O=Organization.

5.4. Need for more studies on organizational level themes of online learning

In this review there is a greater concentration of studies focused on Learner domain topics, and reduced attention to broader more encompassing research themes that fall into the Course and Organization domains. There is a need for organizational level topics such as Access, Culture, Equity, Inclusion and Ethics, and Leadership, Policy and Management to be researched on within the context of online learning. Examination of access, culture, equity, inclusion and ethics is very important to support diverse online learners, particularly with the rapid expansion of online learning across all educational levels. This was also least studied based on Berge and Mrozowski (2001) systematic review.

The topics on leadership, policy and management were least studied both in this review and also in the Tallent-Runnels et al. (2006) and Zawacki-Richter et al. (2009) study. Tallent-Runnels categorized institutional and administrative aspects into institutional policies, institutional support, and enrollment effects. While we included support as a separate category, in this study leadership, policy and management were combined. There is still a need for research on leadership of those who manage online learning, policies for online education, and managing online programs. In the Zawacki-Richter et al. (2009) study, only a few studies examined management and organization focused topics. They also found management and organization to be strongly correlated with costs and benefits. In our study, costs and benefits were collectively included as an aspect of management and organization and not as a theme by itself. These studies will provide research-based evidence for online education administrators.

6. Limitations

As with any systematic review, there are limitations to the scope of the review. The search is limited to twelve journals in the field that typically include research on online learning. These manuscripts were identified by searching the Education Research Complete database which focuses on education students, professionals, and policymakers. Other discipline-specific journals as well as dissertations and proceedings were not included due to the volume of articles. Also, the search was performed using five search terms “online learning" OR "online teaching" OR "online program" OR "online course" OR “online education” in title and keyword. If authors did not include these terms, their respective work may have been excluded from this review even if it focused on online learning. While these terms are commonly used in North America, it may not be commonly used in other parts of the world. Additional studies may exist outside this scope.

The search strategy also affected how we presented results and introduced limitations regarding generalization. We identified that only 8% of the articles published in these journals were related to online learning; however, given the use of search terms to identify articles within select journals it was not feasible to identify the total number of research-based articles in the population. Furthermore, our review focused on the topics and general methods of research and did not systematically consider the quality of the published research. Lastly, some journals may have preferences for publishing studies on a particular topic or that use a particular method (e.g., quantitative methods), which introduces possible selection and publication biases which may skew the interpretation of results due to over/under representation. Future studies are recommended to include more journals to minimize the selection bias and obtain a more representative sample.

Certain limitations can be attributed to the coding process. Overall, the coding process for this review worked well for most articles, as each tended to have an individual or dominant focus as described in the abstracts, though several did mention other categories which likely were simultaneously considered to a lesser degree. However, in some cases, a dominant theme was not as apparent and an effort to create mutually exclusive groups for clearer interpretation the coders were occasionally forced to choose between two categories. To facilitate this coding, the full-texts were used to identify a study focus through a consensus seeking discussion among all authors. Likewise, some studies focused on topics that we have associated with a particular domain, but the design of the study may have promoted an aggregated examination or integrated factors from multiple domains (e.g., engagement). Due to our reliance on author descriptions, the impact of construct validity is likely a concern that requires additional exploration. Our final grouping of codes may not have aligned with the original author's description in the abstract. Additionally, coding of broader constructs which disproportionately occur in the Learner domain, such as learner outcomes, learner characteristics, and engagement, likely introduced bias towards these codes when considering studies that involved multiple domains. Additional refinement to explore the intersection of domains within studies is needed.

7. Implications and future research

One of the strengths of this review is the research categories we have identified. We hope these categories will support future researchers and identify areas and levels of need for future research. Overall, there is some agreement on research themes on online learning research among previous reviews and this one, at the same time there are some contradicting findings. We hope the most-researched themes and least-researched themes provide authors a direction on the importance of research and areas of need to focus on.

The leading themes found in this review is online engagement research. However, presentation of this research was inconsistent, and often lacked specificity. This is not unique to online environments, but the nuances of defining engagement in an online environment are unique and therefore need further investigation and clarification. This review points to seven distinct classifications of online engagement. Further research on engagement should indicate which type of engagement is sought. This level of specificity is necessary to establish instruments for measuring engagement and ultimately testing frameworks for classifying engagement and promoting it in online environments. Also, it might be of importance to examine the relationship between these seven sub-themes of engagement.

Additionally, this review highlights growing attention to learner characteristics, which constitutes a shift in focus away from instructional characteristics and course design. Although this is consistent with the focus on engagement, the role of the instructor, and course design with respect to these outcomes remains important. Results of the learner characteristics and engagement research paired with course design will have important ramifications for the use of teaching and learning professionals who support instruction. Additionally, the review also points to a concentration of research in the area of higher education. With an immediate and growing emphasis on online learning in K-12 and corporate settings, there is a critical need for further investigation in these settings.

Lastly, because the present review did not focus on the overall effect of interventions, opportunities exist for dedicated meta-analyses. Particular attention to research on engagement and learner characteristics as well as how these vary by study design and outcomes would be logical additions to the research literature.

8. Conclusion

This systematic review builds upon three previous reviews which tackled the topic of online learning between 1990 and 2010 by extending the timeframe to consider the most recent set of published research. Covering the most recent decade, our review of 619 articles from 12 leading online learning journal points to a more concentrated focus on the learner domain including engagement and learner characteristics, with more limited attention to topics pertaining to the classroom or organizational level. The review highlights an opportunity for the field to clarify terminology concerning online learning research, particularly in the areas of learner outcomes where there is a tendency to classify research more generally (e.g., engagement). Using this sample of published literature, we provide a possible taxonomy for categorizing this research using subcategories. The field could benefit from a broader conversation about how these categories can shape a comprehensive framework for online learning research. Such efforts will enable the field to effectively prioritize research aims over time and synthesize effects.

Credit author statement

Florence Martin: Conceptualization; Writing - original draft, Writing - review & editing Preparation, Supervision, Project administration. Ting Sun: Methodology, Formal analysis, Writing - original draft, Writing - review & editing. Carl Westine: Methodology, Formal analysis, Writing - original draft, Writing - review & editing, Supervision

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

1 Includes articles that are cited in this manuscript and also included in the systematic review. The entire list of 619 articles used in the systematic review can be obtained by emailing the authors.*

Appendix B Supplementary data to this article can be found online at https://doi.org/10.1016/j.compedu.2020.104009 .

Appendix A. 

Research Themes by the Settings in the Online Learning Publications

Research Themes by the Methodology in the Online Learning Publications

Appendix B. Supplementary data

The following are the Supplementary data to this article:

References 1

• Ahn J., Butler B.S., Alam A., Webster S.A. Learner participation and engagement in open online courses: Insights from the Peer 2 Peer University. MERLOT Journal of Online Learning and Teaching. 2013; 9 (2):160–171. * [ Google Scholar ]
• Akcaoglu M., Lee E. Increasing social presence in online learning through small group discussions. International Review of Research in Open and Distance Learning. 2016; 17 (3) * [ Google Scholar ]
• Allen I.E., Seaman J. Babson survey research group; 2017. Digital compass learning: Distance education enrollment Report 2017. [ Google Scholar ]
• Amador J.A., Mederer H. Migrating successful student engagement strategies online: Opportunities and challenges using jigsaw groups and problem-based learning. Journal of Online Learning and Teaching. 2013; 9 (1):89. * [ Google Scholar ]
• Anderson L.W., Bourke S.F. Routledge; 2013. Assessing affective characteristics in the schools. [ Google Scholar ]
• Archibald D. Fostering the development of cognitive presence: Initial findings using the community of inquiry survey instrument. The Internet and Higher Education. 2010; 13 (1–2):73–74. * [ Google Scholar ]
• Artino A.R., Jr., Stephens J.M. Academic motivation and self-regulation: A comparative analysis of undergraduate and graduate students learning online. The Internet and Higher Education. 2009; 12 (3–4):146–151. [ Google Scholar ]
• Barnard L., Lan W.Y., To Y.M., Paton V.O., Lai S.L. Measuring self-regulation in online and blended learning environments. Internet and Higher Education. 2009; 12 (1):1–6. * [ Google Scholar ]
• Bayeck R.Y., Hristova A., Jablokow K.W., Bonafini F. Exploring the relevance of single‐gender group formation: What we learn from a massive open online course (MOOC) British Journal of Educational Technology. 2018; 49 (1):88–100. * [ Google Scholar ]
• Berge Z., Mrozowski S. Review of research in distance education, 1990 to 1999. American Journal of Distance Education. 2001; 15 (3):5–19. doi: 10.1080/08923640109527090. [ CrossRef ] [ Google Scholar ]
• Berry S. Building community in online doctoral classrooms: Instructor practices that support community. Online Learning. 2017; 21 (2):n2. * [ Google Scholar ]
• Boling E.C., Holan E., Horbatt B., Hough M., Jean-Louis J., Khurana C., Spiezio C. Using online tools for communication and collaboration: Understanding educators' experiences in an online course. The Internet and Higher Education. 2014; 23 :48–55. * [ Google Scholar ]
• Bolliger D.U., Inan F.A. Development and validation of the online student connectedness survey (OSCS) International Review of Research in Open and Distance Learning. 2012; 13 (3):41–65. * [ Google Scholar ]
• Bradford G., Wyatt S. Online learning and student satisfaction: Academic standing, ethnicity and their influence on facilitated learning, engagement, and information fluency. The Internet and Higher Education. 2010; 13 (3):108–114. * [ Google Scholar ]
• Broadbent J. Comparing online and blended learner's self-regulated learning strategies and academic performance. The Internet and Higher Education. 2017; 33 :24–32. [ Google Scholar ]
• Buzdar M., Ali A., Tariq R. Emotional intelligence as a determinant of readiness for online learning. International Review of Research in Open and Distance Learning. 2016; 17 (1) * [ Google Scholar ]
• Capdeferro N., Romero M., Barberà E. Polychronicity: Review of the literature and a new configuration for the study of this hidden dimension of online learning. Distance Education. 2014; 35 (3):294–310. [ Google Scholar ]
• Chaiprasurt C., Esichaikul V. Enhancing motivation in online courses with mobile communication tool support: A comparative study. International Review of Research in Open and Distance Learning. 2013; 14 (3):377–401. [ Google Scholar ]
• Chen C.H., Wu I.C. The interplay between cognitive and motivational variables in a supportive online learning system for secondary physical education. Computers & Education. 2012; 58 (1):542–550. * [ Google Scholar ]
• Cho H. Under co-construction: An online community of practice for bilingual pre-service teachers. Computers & Education. 2016; 92 :76–89. * [ Google Scholar ]
• Cho M.H., Shen D. Self-regulation in online learning. Distance Education. 2013; 34 (3):290–301. [ Google Scholar ]
• Cole M.T., Shelley D.J., Swartz L.B. Online instruction, e-learning, and student satisfaction: A three-year study. International Review of Research in Open and Distance Learning. 2014; 15 (6) * [ Google Scholar ]
• Comer D.K., Clark C.R., Canelas D.A. Writing to learn and learning to write across the disciplines: Peer-to-peer writing in introductory-level MOOCs. International Review of Research in Open and Distance Learning. 2014; 15 (5):26–82. * [ Google Scholar ]
• Cundell A., Sheepy E. Student perceptions of the most effective and engaging online learning activities in a blended graduate seminar. Online Learning. 2018; 22 (3):87–102. * [ Google Scholar ]
• Cung B., Xu D., Eichhorn S. Increasing interpersonal interactions in an online course: Does increased instructor email activity and voluntary meeting time in a physical classroom facilitate student learning? Online Learning. 2018; 22 (3):193–215. [ Google Scholar ]
• Cunningham U.M., Fägersten K.B., Holmsten E. Can you hear me, Hanoi?" Compensatory mechanisms employed in synchronous net-based English language learning. International Review of Research in Open and Distance Learning. 2010; 11 (1):161–177. [ Google Scholar ]
• Davis D., Chen G., Hauff C., Houben G.J. Activating learning at scale: A review of innovations in online learning strategies. Computers & Education. 2018; 125 :327–344. [ Google Scholar ]
• Delen E., Liew J., Willson V. Effects of interactivity and instructional scaffolding on learning: Self-regulation in online video-based environments. Computers & Education. 2014; 78 :312–320. [ Google Scholar ]
• Dixson M.D. Measuring student engagement in the online course: The Online Student Engagement scale (OSE) Online Learning. 2015; 19 (4):n4. * [ Google Scholar ]
• Dray B.J., Lowenthal P.R., Miszkiewicz M.J., Ruiz‐Primo M.A., Marczynski K. Developing an instrument to assess student readiness for online learning: A validation study. Distance Education. 2011; 32 (1):29–47. * [ Google Scholar ]
• Dziuban C., Moskal P., Thompson J., Kramer L., DeCantis G., Hermsdorfer A. Student satisfaction with online learning: Is it a psychological contract? Online Learning. 2015; 19 (2):n2. * [ Google Scholar ]
• Ergün E., Usluel Y.K. An analysis of density and degree-centrality according to the social networking structure formed in an online learning environment. Journal of Educational Technology & Society. 2016; 19 (4):34–46. * [ Google Scholar ]
• Esfijani A. Measuring quality in online education: A meta-synthesis. American Journal of Distance Education. 2018; 32 (1):57–73. [ Google Scholar ]
• Glazer H.R., Murphy J.A. Optimizing success: A model for persistence in online education. American Journal of Distance Education. 2015; 29 (2):135–144. [ Google Scholar ]
• Glazer H.R., Wanstreet C.E. Connection to the academic community: Perceptions of students in online education. Quarterly Review of Distance Education. 2011; 12 (1):55. * [ Google Scholar ]
• Hartnett M., George A.S., Dron J. Examining motivation in online distance learning environments: Complex, multifaceted and situation-dependent. International Review of Research in Open and Distance Learning. 2011; 12 (6):20–38. [ Google Scholar ]
• Harwell M.R. 2012. Research design in qualitative/quantitative/mixed methods. Section III. Opportunities and challenges in designing and conducting inquiry. [ Google Scholar ]
• Hung J.L. Trends of e‐learning research from 2000 to 2008: Use of text mining and bibliometrics. British Journal of Educational Technology. 2012; 43 (1):5–16. [ Google Scholar ]
• Jiang W. Interdependence of roles, role rotation, and sense of community in an online course. Distance Education. 2017; 38 (1):84–105. [ Google Scholar ]
• Ke F., Kwak D. Online learning across ethnicity and age: A study on learning interaction participation, perception, and learning satisfaction. Computers & Education. 2013; 61 :43–51. [ Google Scholar ]
• Kent M. Changing the conversation: Facebook as a venue for online class discussion in higher education. MERLOT Journal of Online Learning and Teaching. 2013; 9 (4):546–565. * [ Google Scholar ]
• Kim C., Park S.W., Cozart J. Affective and motivational factors of learning in online mathematics courses. British Journal of Educational Technology. 2014; 45 (1):171–185. [ Google Scholar ]
• Kizilcec R.F., Pérez-Sanagustín M., Maldonado J.J. Self-regulated learning strategies predict learner behavior and goal attainment in Massive Open Online Courses. Computers & Education. 2017; 104 :18–33. [ Google Scholar ]
• Kopp B., Matteucci M.C., Tomasetto C. E-tutorial support for collaborative online learning: An explorative study on experienced and inexperienced e-tutors. Computers & Education. 2012; 58 (1):12–20. [ Google Scholar ]
• Koseoglu S., Doering A. Understanding complex ecologies: An investigation of student experiences in adventure learning programs. Distance Education. 2011; 32 (3):339–355. * [ Google Scholar ]
• Kumi-Yeboah A. Designing a cross-cultural collaborative online learning framework for online instructors. Online Learning. 2018; 22 (4):181–201. * [ Google Scholar ]
• Kuo Y.C., Walker A.E., Belland B.R., Schroder K.E. A predictive study of student satisfaction in online education programs. International Review of Research in Open and Distance Learning. 2013; 14 (1):16–39. * [ Google Scholar ]
• Kuo Y.C., Walker A.E., Schroder K.E., Belland B.R. Interaction, Internet self-efficacy, and self-regulated learning as predictors of student satisfaction in online education courses. Internet and Higher Education. 2014; 20 :35–50. * [ Google Scholar ]
• Lee J. An exploratory study of effective online learning: Assessing satisfaction levels of graduate students of mathematics education associated with human and design factors of an online course. International Review of Research in Open and Distance Learning. 2014; 15 (1) [ Google Scholar ]
• Lee S.M. The relationships between higher order thinking skills, cognitive density, and social presence in online learning. The Internet and Higher Education. 2014; 21 :41–52. * [ Google Scholar ]
• Lee K. Rethinking the accessibility of online higher education: A historical review. The Internet and Higher Education. 2017; 33 :15–23. [ Google Scholar ]
• Lee Y., Choi J. A review of online course dropout research: Implications for practice and future research. Educational Technology Research & Development. 2011; 59 (5):593–618. [ Google Scholar ]
• Li L.Y., Tsai C.C. Accessing online learning material: Quantitative behavior patterns and their effects on motivation and learning performance. Computers & Education. 2017; 114 :286–297. [ Google Scholar ]
• Liyanagunawardena T., Adams A., Williams S. MOOCs: A systematic study of the published literature 2008-2012. International Review of Research in Open and Distance Learning. 2013; 14 (3):202–227. [ Google Scholar ]
• Lowes S., Lin P., Kinghorn B.R. Gender differences in online high school courses. Online Learning. 2016; 20 (4):100–117. [ Google Scholar ]
• Marbouti F., Wise A.F. Starburst: A new graphical interface to support purposeful attention to others' posts in online discussions. Educational Technology Research & Development. 2016; 64 (1):87–113. * [ Google Scholar ]
• Martin F., Ahlgrim-Delzell L., Budhrani K. Systematic review of two decades (1995 to 2014) of research on synchronous online learning. American Journal of Distance Education. 2017; 31 (1):3–19. [ Google Scholar ]
• Moore-Adams B.L., Jones W.M., Cohen J. Learning to teach online: A systematic review of the literature on K-12 teacher preparation for teaching online. Distance Education. 2016; 37 (3):333–348. [ Google Scholar ]
• Murphy E., Rodríguez-Manzanares M.A. Rapport in distance education. International Review of Research in Open and Distance Learning. 2012; 13 (1):167–190. * [ Google Scholar ]
• Nye A. Building an online academic learning community among undergraduate students. Distance Education. 2015; 36 (1):115–128. * [ Google Scholar ]
• Olesova L., Slavin M., Lim J. Exploring the effect of scripted roles on cognitive presence in asynchronous online discussions. Online Learning. 2016; 20 (4):34–53. * [ Google Scholar ]
• Orcutt J.M., Dringus L.P. Beyond being there: Practices that establish presence, engage students and influence intellectual curiosity in a structured online learning environment. Online Learning. 2017; 21 (3):15–35. * [ Google Scholar ]
• Overbaugh R.C., Nickel C.E. A comparison of student satisfaction and value of academic community between blended and online sections of a university-level educational foundations course. The Internet and Higher Education. 2011; 14 (3):164–174. * [ Google Scholar ]
• O'Shea S., Stone C., Delahunty J. “I ‘feel’like I am at university even though I am online.” Exploring how students narrate their engagement with higher education institutions in an online learning environment. Distance Education. 2015; 36 (1):41–58. * [ Google Scholar ]
• Paechter M., Maier B. Online or face-to-face? Students' experiences and preferences in e-learning. Internet and Higher Education. 2010; 13 (4):292–297. [ Google Scholar ]
• Phirangee K. Students' perceptions of learner-learner interactions that weaken a sense of community in an online learning environment. Online Learning. 2016; 20 (4):13–33. * [ Google Scholar ]
• Phirangee K., Malec A. Othering in online learning: An examination of social presence, identity, and sense of community. Distance Education. 2017; 38 (2):160–172. * [ Google Scholar ]
• Preisman K.A. Teaching presence in online education: From the instructor's point of view. Online Learning. 2014; 18 (3):n3. * [ Google Scholar ]
• Rowe M. Developing graduate attributes in an open online course. British Journal of Educational Technology. 2016; 47 (5):873–882. * [ Google Scholar ]
• Ruane R., Koku E.F. Social network analysis of undergraduate education student interaction in online peer mentoring settings. Journal of Online Learning and Teaching. 2014; 10 (4):577–589. * [ Google Scholar ]
• Ruane R., Lee V.J. Analysis of discussion board interaction in an online peer mentoring site. Online Learning. 2016; 20 (4):79–99. * [ Google Scholar ]
• Rye S.A., Støkken A.M. The implications of the local context in global virtual education. International Review of Research in Open and Distance Learning. 2012; 13 (1):191–206. * [ Google Scholar ]
• Saadatmand M., Kumpulainen K. Participants' perceptions of learning and networking in connectivist MOOCs. Journal of Online Learning and Teaching. 2014; 10 (1):16. * [ Google Scholar ]
• Shackelford J.L., Maxwell M. Sense of community in graduate online education: Contribution of learner to learner interaction. International Review of Research in Open and Distance Learning. 2012; 13 (4):228–249. * [ Google Scholar ]
• Shea P., Bidjerano T. Does online learning impede degree completion? A national study of community college students. Computers & Education. 2014; 75 :103–111. * [ Google Scholar ]
• Sherry L. Issues in distance learning. International Journal of Educational Telecommunications. 1996; 1 (4):337–365. [ Google Scholar ]
• Slagter van Tryon P.J., Bishop M.J. Evaluating social connectedness online: The design and development of the social perceptions in learning contexts instrument. Distance Education. 2012; 33 (3):347–364. * [ Google Scholar ]
• Swaggerty E.A., Broemmel A.D. Authenticity, relevance, and connectedness: Graduate students' learning preferences and experiences in an online reading education course. The Internet and Higher Education. 2017; 32 :80–86. * [ Google Scholar ]
• Tallent-Runnels M.K., Thomas J.A., Lan W.Y., Cooper S., Ahern T.C., Shaw S.M., Liu X. Teaching courses online: A review of the research. Review of Educational Research. 2006; 76 (1):93–135. doi: 10.3102/00346543076001093. [ CrossRef ] [ Google Scholar ]
• Tawfik A.A., Giabbanelli P.J., Hogan M., Msilu F., Gill A., York C.S. Effects of success v failure cases on learner-learner interaction. Computers & Education. 2018; 118 :120–132. [ Google Scholar ]
• Thomas J. Exploring the use of asynchronous online discussion in health care education: A literature review. Computers & Education. 2013; 69 :199–215. [ Google Scholar ]
• Thormann J., Fidalgo P. Guidelines for online course moderation and community building from a student's perspective. Journal of Online Learning and Teaching. 2014; 10 (3):374–388. * [ Google Scholar ]
• Tibi M.H. Computer science students' attitudes towards the use of structured and unstructured discussion forums in fully online courses. Online Learning. 2018; 22 (1):93–106. * [ Google Scholar ]
• Tsai C.W., Chiang Y.C. Research trends in problem‐based learning (pbl) research in e‐learning and online education environments: A review of publications in SSCI‐indexed journals from 2004 to 2012. British Journal of Educational Technology. 2013; 44 (6):E185–E190. [ Google Scholar ]
• Tsai C.W., Fan Y.T. Research trends in game‐based learning research in online learning environments: A review of studies published in SSCI‐indexed journals from 2003 to 2012. British Journal of Educational Technology. 2013; 44 (5):E115–E119. [ Google Scholar ]
• Tsai C.W., Shen P.D., Chiang Y.C. Research trends in meaningful learning research on e‐learning and online education environments: A review of studies published in SSCI‐indexed journals from 2003 to 2012. British Journal of Educational Technology. 2013; 44 (6):E179–E184. [ Google Scholar ]
• Tsai C.W., Shen P.D., Fan Y.T. Research trends in self‐regulated learning research in online learning environments: A review of studies published in selected journals from 2003 to 2012. British Journal of Educational Technology. 2013; 44 (5):E107–E110. [ Google Scholar ]
• U.S. Department of Education, Institute of Education Sciences . InstituteofEducationSciences; Washington,DC: 2017. What Works Clearinghouse procedures and standards handbook, version3.0. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_v3_0_standards_handbook.pdf Retrievedfrom. [ Google Scholar ]
• Veletsianos G., Shepherdson P. A systematic analysis and synthesis of the empirical MOOC literature published in 2013–2015. International Review of Research in Open and Distance Learning. 2016; 17 (2) [ Google Scholar ]
• VERBI Software . 2019. MAXQDA 2020 online manual. Retrieved from maxqda. Com/help-max20/welcome [ Google Scholar ]
• Verstegen D., Dailey-Hebert A., Fonteijn H., Clarebout G., Spruijt A. How do virtual teams collaborate in online learning tasks in a MOOC? International Review of Research in Open and Distance Learning. 2018; 19 (4) * [ Google Scholar ]
• Wang Y., Baker R. Grit and intention: Why do learners complete MOOCs? International Review of Research in Open and Distance Learning. 2018; 19 (3) * [ Google Scholar ]
• Wei C.W., Chen N.S., Kinshuk A model for social presence in online classrooms. Educational Technology Research & Development. 2012; 60 (3):529–545. * [ Google Scholar ]
• Wicks D., Craft B.B., Lee D., Lumpe A., Henrikson R., Baliram N., Wicks K. An evaluation of low versus high collaboration in online learning. Online Learning. 2015; 19 (4):n4. * [ Google Scholar ]
• Wise A.F., Perera N., Hsiao Y.T., Speer J., Marbouti F. Microanalytic case studies of individual participation patterns in an asynchronous online discussion in an undergraduate blended course. The Internet and Higher Education. 2012; 15 (2):108–117. * [ Google Scholar ]
• Wisneski J.E., Ozogul G., Bichelmeyer B.A. Does teaching presence transfer between MBA teaching environments? A comparative investigation of instructional design practices associated with teaching presence. The Internet and Higher Education. 2015; 25 :18–27. * [ Google Scholar ]
• Wladis C., Hachey A.C., Conway K. An investigation of course-level factors as predictors of online STEM course outcomes. Computers & Education. 2014; 77 :145–150. * [ Google Scholar ]
• Wladis C., Samuels J. Do online readiness surveys do what they claim? Validity, reliability, and subsequent student enrollment decisions. Computers & Education. 2016; 98 :39–56. [ Google Scholar ]
• Yamagata-Lynch L.C. Blending online asynchronous and synchronous learning. International Review of Research in Open and Distance Learning. 2014; 15 (2) * [ Google Scholar ]
• Yang J., Kinshuk, Yu H., Chen S.J., Huang R. Strategies for smooth and effective cross-cultural online collaborative learning. Journal of Educational Technology & Society. 2014; 17 (3):208–221. * [ Google Scholar ]
• Yeboah A.K., Smith P. Relationships between minority students online learning experiences and academic performance. Online Learning. 2016; 20 (4):n4. * [ Google Scholar ]
• Yu T. Examining construct validity of the student online learning readiness (SOLR) instrument using confirmatory factor analysis. Online Learning. 2018; 22 (4):277–288. * [ Google Scholar ]
• Yukselturk E., Bulut S. Gender differences in self-regulated online learning environment. Educational Technology & Society. 2009; 12 (3):12–22. [ Google Scholar ]
• Yukselturk E., Top E. Exploring the link among entry characteristics, participation behaviors and course outcomes of online learners: An examination of learner profile using cluster analysis. British Journal of Educational Technology. 2013; 44 (5):716–728. [ Google Scholar ]
• Zawacki-Richter O., Backer E., Vogt S. Review of distance education research (2000 to 2008): Analysis of research areas, methods, and authorship patterns. International Review of Research in Open and Distance Learning. 2009; 10 (6):30. doi: 10.19173/irrodl.v10i6.741. [ CrossRef ] [ Google Scholar ]
• Zhu M., Sari A., Lee M.M. A systematic review of research methods and topics of the empirical MOOC literature (2014–2016) The Internet and Higher Education. 2018; 37 :31–39. [ Google Scholar ]
• Zimmerman T.D. Exploring learner to content interaction as a success factor in online courses. International Review of Research in Open and Distance Learning. 2012; 13 (4):152–165. [ Google Scholar ]

Professional teacher training in the PELC and PST social programs in Brazil: A systematic review

This research aimed to analyze the scientific production on public sports and leisure policies related to professional training of two programs in Brazil: City Sport and Leisure Program (CSLP) and Second Time Program (STP). As a methodological procedure, a systematic review with a qualitative approach was carried out between the years 2003 and 2022. The research sample was composed of 21 articles, published between 2012 and 2022. The results showed that researchers from the South and Southeast region boosted publications on professional training in CSLP and STP. Paper publications were concentrated in journals in the fields of physical education, education, sport and leisure. Professional training corresponds to a central element for structuring sports social programs, as a way to minimize gaps in initial training, but the format of sporadic courses and distance from the reality of professionals was the main limiting factor.

Author Biographies

Dirceu santos silva, universidade federal de mato grosso do sul (ufms).

Doutor em Educação Física pela Unicamp. Professor do Programa de Pós-graduação em Educação da Universidade Federal de Mato Grosso do Sul.

Luana Carla André, Universidade Federal de Mato Grosso do Sul (UFMS)

Mestre em educação pelo Programa de Pós-graduação em Educação da Universidade Federal de Mato Grosso do Sul.

Mariana Brasiliano Salerno, Universidade Federal de Mato Grosso do Sul (UFMS)

Doutora em Educação Física pela Unicamp. Professor do Programa de Pós-graduação em Educação da Universidade Federal de Mato Grosso do Sul.

• PDF (Português)

How to Cite

• Endnote/Zotero/Mendeley (RIS)

Copyright (c) 2024 Dirceu Santos Silva, Luana Carla André, Mariana Brasiliano Salerno

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License .

Education Policy Analysis Archives/Archivos Analíticos de Políticas Educativas/Arquivos Analíticos de Políticas Educativas (EPAA/AAPE;  ISSN 1068-2341 ) is a peer-reviewed, open-access, international, multilingual, and multidisciplinary journal designed for researchers, practitioners, policy makers, and development analysts concerned with education policies.

• Português (Brasil)
• Español (España)

Make a Gift

Submit a manuscript or revision, recent special issues.

Youth and Adult Education, Literacies and Decoloniality

Global Policy Mobilities in Federal Education Systems

Teacher Subjectivities in Latin America

• Open access
• Published: 12 May 2024

Distance plus attention for binding affinity prediction

• Julia Rahman 1   na1 ,
• M. A. Hakim Newton 2 , 3   na1 ,
• Mohammed Eunus Ali 4 &
• Abdul Sattar 2  

Journal of Cheminformatics volume  16 , Article number:  52 ( 2024 ) Cite this article

200 Accesses

Metrics details

Protein-ligand binding affinity plays a pivotal role in drug development, particularly in identifying potential ligands for target disease-related proteins. Accurate affinity predictions can significantly reduce both the time and cost involved in drug development. However, highly precise affinity prediction remains a research challenge. A key to improve affinity prediction is to capture interactions between proteins and ligands effectively. Existing deep-learning-based computational approaches use 3D grids, 4D tensors, molecular graphs, or proximity-based adjacency matrices, which are either resource-intensive or do not directly represent potential interactions. In this paper, we propose atomic-level distance features and attention mechanisms to capture better specific protein-ligand interactions based on donor-acceptor relations, hydrophobicity, and \(\pi \) -stacking atoms. We argue that distances encompass both short-range direct and long-range indirect interaction effects while attention mechanisms capture levels of interaction effects. On the very well-known CASF-2016 dataset, our proposed method, named Distance plus Attention for Affinity Prediction (DAAP), significantly outperforms existing methods by achieving Correlation Coefficient (R) 0.909, Root Mean Squared Error (RMSE) 0.987, Mean Absolute Error (MAE) 0.745, Standard Deviation (SD) 0.988, and Concordance Index (CI) 0.876. The proposed method also shows substantial improvement, around 2% to 37%, on five other benchmark datasets. The program and data are publicly available on the website https://gitlab.com/mahnewton/daap.

Scientific Contribution Statement

This study innovatively introduces distance-based features to predict protein-ligand binding affinity, capitalizing on unique molecular interactions. Furthermore, the incorporation of protein sequence features of specific residues enhances the model’s proficiency in capturing intricate binding patterns. The predictive capabilities are further strengthened through the use of a deep learning architecture with attention mechanisms, and an ensemble approach, averaging the outputs of five models, is implemented to ensure robust and reliable predictions.

Introduction

Conventional drug discovery, as noted by a recent study [ 1 ], is a resource-intensive and time-consuming process that typically lasts for about 10 to 15 years and costs approximately 2.558 billion USD to bring each new drug successfully to the market. Computational approaches can expedite the drug discovery process by identifying drug molecules or ligands that have high binding affinities towards disease-related proteins and would thus form strong transient bonds to inhibit protein functions [ 2 , 3 , 4 ]. In a typical drug development pipeline, a pool of potential ligands is usually given, and the ligands exhibiting strong binding affinities are identified as the most promising drug candidates against a target protein. In essence, protein-ligand binding affinity values serve as a scoring method to narrow the search space for virtual screening [ 5 ].

Existing computational methods for protein-ligand binding affinity prediction include both traditional machine learning and deep learning-based approaches. Early methods used Kernel Partial Least Squares [ 6 ], Support Vector Regression (SVR) [ 7 ], Random Forest (RF) Regression [ 8 ], and Gradient Boosting [ 9 ]. However, just like various other domains [ 10 , 11 , 12 , 13 , 14 ], drug discovery has also seen significant recent advancements [ 15 , 16 , 17 , 18 ] from the computational power and extensive datasets used in deep learning. Deep learning models for protein-ligand binding affinity prediction take protein-ligand docked complexes as input and give binding affinity values as output. Moreover, these models use various input features to capture the global characteristics of the proteins and the ligands and their local interactions in the pocket areas where the ligands get docked into the proteins.

Recent deep learning models for protein-ligand binding affinity prediction include DeepDTA [ 19 ], Pafnucy [ 20 ], \(K_\text {DEEP}\) [ 21 ], DeepAtom [ 22 ], DeepDTAF [ 23 ], BAPA [ 5 ], SFCNN [ 24 ], DLSSAffinity [ 4 ] EGNA [ 25 ], CAPLA [ 26 ] and ResBiGAAT [ 27 ]. DeepDTA [ 19 ] introduced a Convolutional Neural Network (CNN) model with input features Simplified Molecular Input Line Entry System (SMILES) sequences for ligands and full-length protein sequences. Pafnucy and \(K_{DEEP}\) used a 3D-CNN with 4D tensor representations of the protein-ligand complexes as input features. DeepAtom employed a 3D-CNN to automatically extract binding-related atomic interaction patterns from voxelized complex structures. DeepDTAF combined global contextual features and local binding area-related features with dilated convolution to capture multiscale long-range interactions. BAPA introduced a deep neural network model for affinity prediction, featuring descriptor embeddings and an attention mechanism to capture local structural details. SFCNN employed a 3D-CNN with simplified 4D tensor features having only basic atomic type information. DLSSAffinity employed 1D-CNN with pocket-ligand structural pairs as local features and ligand SMILES and protein sequences as global features. EGNA introduced an empirical graph neural network (GNN) that utilizes graphs to represent proteins, ligands, and their interactions in the pocket areas. CAPLA [ 26 ] utilized a cross-attention mechanism within a CNN along with sequence-level input features for proteins and ligands and structural features for secondary structural elements. ResBiGAAT [ 27 ] integrates a deep Residual Bidirectional Gated Recurrent Unit (Bi-GRU) with two-sided self-attention mechanisms, utilizing both protein and ligand sequence-level features along with their physicochemical properties for efficient prediction of protein-ligand binding affinity.

In this work, we consider the effective capturing of protein-ligand interaction as a key to making further progress in binding affinity prediction. However, as we see from the literature, a sequential feature-based model such as DeepDTA was designed mainly to capture long-range interactions between proteins and ligands, not considering local interactions. CAPLA incorporates cross-attention mechanisms along with sequence-based features to indirectly encompass short-range interactions to some extent. ResBiGAAT employs a residual Bi-GRU architecture and two-sided self-attention mechanisms to capture long-term dependencies between protein and ligand molecules, utilizing SMILES representations, protein sequences, and diverse physicochemical properties for improved binding affinity prediction. On the other hand, structural feature-based models such as Pafnucy, \(K_{DEEP}\) and SFCNN use 3D grids, 4D tensors, or molecular graph representations. These features provide valuable insights into the pocket region of the protein-ligand complexes but incur significant computational costs in terms of memory and processing time. Additionally, these features have limitations in capturing long-range indirect interactions among protein-ligand pairs. DLSSAffinity aims to bridge the gap between short- and long-range interactions by considering both sequential and structural features. Moreover, DLSSAffinity uses 4D tensors for Cartesian coordinates and atom-level features to represent interactions between heavy atoms in the pocket areas of the protein-ligand complexes. These representations of interactions are still indirect, considering the importance of protein-ligand interaction in binding affinity. EGNA tried to use graphs and Boolean-valued adjacency matrices to capture protein-ligand interactions to some extent. However, EGNA’s interaction graph considers only edges between each pair of a \(C_\beta \) atom in the pocket areas of the protein and a heavy atom in the ligand when their distance is below a threshold of \(10\mathring{A}\) .

Inspired by the use of distance measures in protein structure prediction [ 14 , 28 , 29 ], in this work, we employ distance-based input features in protein-ligand binding affinity prediction. To be more specific, we use distances between donor-acceptor [ 30 ], hydrophobic [ 31 , 32 ], and \(\pi \) -stacking [ 31 , 32 ] atoms as interactions between such atoms play crucial roles in protein-ligand binding. These distance measures between various types of atoms could essentially capture more direct and more precise information about protein-ligand interactions than using sequence-based features or various other features representing the pocket areas of the protein-ligand complexes. Moreover, the distance values could more directly capture both short- and long-range interactions than adjacency-based interaction graphs of EGNA or tensor-based pocket area representations of DLSSAffinity. Besides capturing protein-ligand interactions, we also consider only those protein residues with donor, hydrophobic, and \(\pi \) -stacking atoms in this work. Considering only these selective residues is also in contrast with all other methods that use all the protein residues. For ligand representation, we use SMILES strings. After concatenating all input features, we use an attention mechanism to effectively weigh the significance of various input features. Lastly, we enhance the predictive performance of our model by adopting an ensembling approach, averaging the outputs of several trained models.

We name our proposed method as Distance plus Attention for Affinity Prediction (DAAP). On the very well-known CASF-2016 dataset, DAAP significantly outperforms existing methods by achieving the Correlation Coefficient (R) 0.909, Root Mean Squared Error (RMSE) 0.987, Mean Absolute Error (MAE) 0.745, Standard Deviation (SD) 0.988, and Concordance Index (CI) 0.876. DAAP also shows substantial improvement, ranging from 2% to 37%, on five other benchmark datasets. The program and data are publicly available on the website https://gitlab.com/mahnewton/daap .

In our study, we first demonstrate the robustness of our deep architecture through five-fold cross-validation. Subsequently, the learning curve, as depicted in Fig.  1 , illustrates the dynamics of training and validation loss, providing insights into the stability and reliability of the learning process. Furthermore, we provide a comprehensive performance comparison of our proposed model with current state-of-the-art predictors. We also provide an in-depth analysis of the experimental results. The effectiveness of our proposed features is substantiated through an ablation study and a detailed analysis of input features.

Training and validation loss curve of DAAP

Five-fold cross-validation

This study employs a five-fold cross-validation approach to evaluate the performance of the proposed model thoroughly, demonstrating the robustness of the deep architecture. Table   1 provides the average performance metrics (R, RMSE, MAE, SD, and CI) along with their corresponding standard deviations derived from the 5-fold cross-validation on the CASF \(-\) 2016.290 test set when the model is trained with PDBbind2016 and PDBbind2020 datasets. This presentation highlights the predictor’s predictive accuracy and reliability, emphasising the proposed model’s effectiveness.

Average ensemble

Our proposed approach leverages an attention-based deep learning architecture to predict binding affinity. The input feature set comprises distance matrices, sequence-based features for specific protein residues, and SMILES sequences. To enhance the robustness and mitigate the effects of variability and overfitting, we train five models and employ arithmetic averaging for ensembling. Average ensembling is more suitable than max voting ensembling when dealing with real values.

Table  2 shows the results of five models and their averages when all models have the identical setting of their training parameters and the training datasets. We see that the ensemble results are better than the results of the individual models in both the PDBbind2016 and PDBbind2020 training datasets. To check that the proposed approach is robust over the variability in the training datasets, we also train five models but each with a different training subset. These training subsets were obtained by using sampling with replacement. Table  3 shows the results of these five models and their averages.

Tables  2 and 3 depict that the ensemble results are better than the results of the individual results in both training sets. It might seem counterintuitive to see the average results are better than all the individual results, but note that these are not simple average of averages. When the ensemble results are compared across Tables  2 and 3 , the best results are observed in Table  2 for the PDBbind2020 training set. All evaluation metrics R, RMSE, SD, MAE, and CI display improved performance when using the same training data (Table  2 ) compared to different varying training data (Table  3 ) in PDBbind2020 data set. Accordingly, we choose the ensemble with the same training data for PDBbind2020 (Table  2 ) as our final binding affinity prediction model. Conversely, for PDBbind2016, superior outcomes are obtained from the varied training subsets in Table  3 . Henceforth, the best-performing models using PDBbind2016 and PDBbind2020 will be referred to as DAAP16 and DAAP20, respectively, in subsequent discussions.

Comparison with state-of-the-art methods

In our comparative analysis, we assess the performance of our proposed affinity predictor, DAAP, on the CASF-2016 test set, compared to nine recent state-of-the-art predictors: Pafnucy [ 20 ], DeepDTA [ 19 ], OnionNet [ 3 ], DeepDTAF [ 23 ], SFCNN [ 24 ] DLSSAffinity [ 4 ], EGNA [ 25 ], CAPLA [ 26 ] and ResBiGAAT [ 27 ]. Notably, the most recent predictors have surpassed the performance of the initial four, prompting us to focus our comparison on their reported results. For the latter five predictors, we detail the methodology of obtaining their results as follows:

DLSSAffinity We rely on the results available on DLSSAffinity’s GitHub repository, as direct prediction for specific target proteins is not possible due to the unavailability of its trained model.

SFCNN Utilizing the provided weights and prediction code from SFCNN, we replicate their results, except for CASF-2013. The ambiguity regarding the inclusion of CASF-2013 data in their training set (sourced from the PDBbind database version 2019) leads us to omit these from our comparison.

EGNA We have adopted EGNA’s published results for the CASF-2016 test set with 285 protein-ligand complexes due to differing Uniclust30 database versions for HHM feature construction. We applied EGNA’s code with our HHM features for the other five test sets to ensure a consistent evaluation framework.

CAPLA Predictions are made based on the features given in CAPLA’s GitHub, except for the ADS.74 dataset, where we can’t predict results due to the unavailability of feature sets. Their results are the same as their reported results.

ResBiGAAT We included ResBiGAAT’s published results in our analysis after encountering discrepancies with their online server using the same SMILES sequences and protein sequences from test PDB files as us. Variations in results, particularly for PDB files with multiple chains, led us to rely on their reported data, as it yielded more consistent and higher accuracies than our attempts.

In Table  4 , the first 8 methods, namely Pafnucy, DeepDTA, OnionNet, DeepDTAF, DLSSAffinity, SFCNN, \(EGNA^*\) and CAPLA reported on 290 CASF-2016 protein-ligand complexes. To make a fair comparison with these 8 methods, we compared our proposed method DAAP16 and DAAP20 on those 290 protein-ligand complexes. From the data presented in the Table  4 , it is clear that our DAAP20 approach outperforms all the 8 predictors, achieving the highest R-value of 0.909, the highest CI value of 0.876, the lowest RMSE of 0.987, the lowest MAE of 0.745, and the lowest SD of 0.988. Specifically, compared to the closest state-of-the-art predictor, CAPLA, our approach demonstrated significant improvements, with approximately 5% improvement in R, 12% in RMSE, 14% in MAE, 11% in SD, and 4% in CI metrics, showcasing its superior predictive capabilities. As 3 of the recent predictors, namely SFCNN, EGNA, and ResBiGAAT, reported their result for the 285 protein-ligand complexes on the CASF-2016 dataset, to make a fair comparison with them as well, we assess our predictor, DAAP, on these 285 proteins as well. From the data presented in Table  4 , the results revealed that, across all metrics, DAAP20 outperformed these three predictors on 285 proteins as well. Particularly, compared to the recent predictor ResBiGAAT, our approach demonstrated notable improvements, with around 6% improvement in R, 19% in RMSE, 20% in MAE, and 5% in CI metrics, highlighting its superior predictive capabilities.

Table  5 presents a comprehensive evaluation of the prediction performance of our proposed DAAP approach on five other well-known test sets CASF \(-\) 2013.87, CASF \(-\) 2013.195 ADS.74, CSAR-HiQ.51 and CSAR-HiQ.36. Across these test sets, our DAAP approaches demonstrate superior predictive performance in protein-ligand binding affinity. On the CASF \(-\) 2013.87 dataset, EGNA surpasses CAPLA with higher R-value and CI-value of 0.752 and 0.767, respectively, while CAPLA records lower RMSE, MAE and SD values of 1.512, 1.197, and 1.521. In contrast, our DAAP20 surpasses both, excelling in all metrics with an R of 0.811, RMSE of 1.324, MAE of 1.043, SD of 1.332, and CI of 0.813, with DAAP16 also delivering robust performance. For the CASF \(-\) 2013.195 test set, a similar trend is observed with our DAAP20 approach outperforming the nearest state-of-the-art predictor by a significant margin of 8%-20% across all evaluation metrics. The DAAP16 approach, not DAAP20, stands out on the ADS.74 dataset by surpassing predictors like Pafnucy, SFCNN and EGNA, showcasing substantial improvements of approximately 12%-37% in various metrics. When evaluating the CSAR-HiQ.51 and CSAR-HiQ.36 datasets against six state-of-the-art predictors, DAAP20 consistently outperforms all, indicating enhancements of 2%-20% and 3%-31%, respectively. Although DAAP16 does not surpass ResBiGAAT in CSAR-HiQ.51, it notably excels in the CSAR-HiQ.36 dataset, outperforming ResBiGAAT in all metrics except MAE. These results underscore the exceptional predictive capabilities of our DAAP approach across diverse datasets and evaluation criteria, consistently surpassing existing state-of-the-art predictors.

The distributions of real and predicted binding affinity values by our predictor (green) and the closest state-of-the-art predictor (red) across the six test sets

Figure  2 presents the distributions of actual and predicted binding affinities for our best DAAP approach and the closest state-of-the-art predictor. In all six test sets, a clear linear correlation and low mean absolute error (MAE) between predicted and actual binding affinity values can be observed for our DAAP model, demonstrating the strong performance of our model across these test sets. The other predictors show scattering over larger areas. In our analysis, we could not consider ResBiGAAT in the CSAR-HiQ.51 and CSAR-HiQ.36 datasets due to the unavailability of their results.

Ablation study and explainability

A significant contribution of this work is utilising distance matrix input features to capture critical information about the protein-ligand relationship. Specifically, we employ a concatenation of three distance maps, representing donor-acceptor, hydrophobic, and \(\pi \) -stacking interactions, as input features, effectively conveying essential protein-ligand bonding details. Following finalising our prediction architecture by incorporating two additional features derived from protein and SMILES sequences, we conduct an in-depth analysis of the impact of various combinations of these distance matrices as features. In the case of protein features, residues are selected based on which distance maps are considered.

Table  6 illustrates the outcomes obtained from experimenting with different combinations of distance maps and selected protein residue and ligand SMILES features on the CASF \(-\) 2016.290 test set. We devise four unique combinations, employing three distinct distance maps for both the PDBbind2016 and PDBbind2020 training datasets. Additionally, we explore a combination that integrates donor-acceptor, hydrophobic, and \(\pi \) -stacking distance maps with features from all protein residues, denoted as DA + \(\pi \) S + HP + FP, to evaluate the impact of using all residues versus selected ones.

From the information presented in Table  6 , it is evident that utilizing the donor-acceptor (DA) solely distance maps yields the lowest performance across both training sets, particularly when different combinations of distance maps are paired with selective protein residues. However, as expected, the combination of the three distance maps, namely DA, \(\pi \) S ( \(\pi \) -stacking), and HP (Hydrophobicity), demonstrates superior performance compared to other combinations. Notably, the combination of DA and HP outperforms the other two combinations but falls short of our best-performing feature set. The ensemble of DA, \(\pi \) S, HP and all protein residues exhibit the least favourable outcomes among the tested combinations. This result aligns with our expectations, as Hydrophobic interactions are the most prevalent in protein-ligand binding, underscoring their significance in feature analysis.

Integrating an attention mechanism into our model is crucial in achieving improved results. After consolidating the outputs of three 1D-CNN blocks, we employ attention, each receiving inputs from distance maps, protein sequences, and ligand sequences. The dimension of the feature is 384. As depicted in Fig.  3 , the heatmap visualization highlights the differential attention weights assigned to various features, with brighter and darker regions indicating higher weights to certain features, thus improving binding affinity predictions. This process underscores the mechanism’s ability to discern and elevate critical features, showing that not all features are equally important. Further emphasizing the significance of attention, a comparative analysis using the same model architecture without the attention mechanism on the same features-shown in the last row of Table  6 demonstrates its vital role in boosting predictive accuracy. This comparison not only reinforces the value of the attention mechanism in detecting intricate patterns within the feature space but also significantly enhances the model’s predictive capabilities.

Visualization of attention maps for concatenated features in the 1o0h protein-ligand complex of the CASF \(-\) 2016.290 dataset

Statistical analysis

In assessing the statistical significance of performance differences between DAAP and its closest competitors, Wilcoxon Signed Ranked Tests at a 95% confidence level were conducted. Comparisons included DAAP against CAPLA for CASF \(-\) 2016.290, CASF \(-\) 2013.87, CASF \(-\) 2013.195, CSAR-HiQ.36, and CSAR-HiQ.51 datasets and between DAAP and SFCNN for the ADS.74 test set. Unfortunately, ResBiGAAT’s results were unavailable for inclusion in the analysis. Table  7 depicts that DAAP demonstrated statistical significance compared to the closest state-of-the-art predictor across various test sets, as indicated by p-values ranging from 0.000 to 0.047. The consistently negative mean Z-values, ranging from \(-\) 14.71 to \(-\) 5.086, suggest a systematic improvement in predictive performance. Moreover, higher mean rankings, ranging from 19.5 to 144.5, further emphasize the overall superiority of DAAP. Notably, the superior performance is observed across diverse datasets, including CASF \(-\) 2016.290, CASF \(-\) 2013.87, CASF \(-\) 2013.195, ADS.74, CSAR-HiQ.51, and CSAR-HiQ.36. These findings underscore the robustness and effectiveness of DAAP in predicting protein-ligand binding affinity.

Screening results

In this section, we scrutinize the effectiveness of our predicted affinity scores to accurately differentiate between active binders (actives) and non-binders (decoys) throughout the screening procedure. To this end, we have carefully curated a subset of seven hand-verified targets from the Database of Useful Decoys: Enhanced (DUD-E), accessible via https://dude.docking.org , to serve as our evaluative benchmark. The details about seven targets are given in Table  8 . This table underscores the diversity and challenges inherent in the dataset, reflecting a wide range of D/A ratios that present a comprehensive framework for evaluating the discriminatory power of our predicted affinity scores.

To construct protein-ligand complexes for these targets, we employed AutoDock Vina, configuring the docking grid to a \(20\mathring{A} \times 20\mathring{A} \times 20\mathring{A}\) cube centred on the ligand’s position. This setup and 32 consecutive Monte-Carlo sampling iterations identified the optimal pose for each molecule pair. Our evaluation of the screening performance utilizes two pivotal metrics: the Receiver Operating Characteristic (ROC) curve [ 33 ] and the Enrichment Factor (EF) [ 34 ]. Figure  4 shows the ROC curve and the EF graph for a detailed examination of a predictive model’s efficacy in virtual screening. The ROC curve’s analysis, with AUC values spanning from 0.63 to 0.76 for the seven targets, illustrates our model’s proficient capability in differentiating between actives and decoys. These values, closely approaching the top-left corner of the graph, denote a high true positive rate alongside a low false positive rate, underscoring our model’s efficacy.

Screening Performance of the Predictive Model: Roc curve (left) and EF (right)

Furthermore, the EF graph of Fig.  4 provides a quantitative assessment of the model’s success in prioritizing active compounds within the top fractions of the dataset, notably the top 1% to 10%. Initial EF values ranging from 12.3 to 9.9 for the top 1% underscore our model’s exceptional ability to enrich active compounds beyond random chance significantly. This pronounced enrichment highlights the model’s utility in the early identification of promising candidates. However, the observed gradual decline in EF values with increasing dataset fractions aligns with expectations, reflecting the challenge of sustaining high enrichment levels across broader selections.

Conclusions

In our protein-ligand binding affinity prediction, we introduce atomic-level distance map features encompassing donor-acceptor, hydrophobic, and \(\pi \) -stacking interactions, providing deeper insights into interactions for precise predictions, both for short and long-range. We enhance our model further with specific protein sequence features of specific residues and ligand SMILES information. These features are integrated into an attention-based 1D-CNN architecture that is used a number of times for ensemble-based performance enhancement, resulting in superior results compared to existing methods across six benchmark datasets. Remarkably, on the CASF-2016 dataset, our model achieves a Correlation Coefficient (R) of 0.909, Root Mean Squared Error (RMSE) of 0.987, Mean Absolute Error (MAE) of 0.745, Standard Deviation (SD) of 0.988, and Concordance Index (CI) of 0.876, signifying its potential to advance drug discovery binding affinity prediction. The program and data are publicly available on the website https://gitlab.com/mahnewton/daap .

We describe the protein-ligand dataset used in our work. We also describe our proposed method in terms of its input features, output representations, and deep learning architectures.

Protein-ligand datasets

In the domain of protein-ligand binding affinity research, one of the primary sources for training, validation, and test sets is the widely recognized PDBbind database [ 35 ]. This database is meticulously curated. It comprises experimentally verified protein-ligand complexes. Each complex encompasses the three-dimensional structures of a protein-ligand pair alongside its corresponding binding affinities expressed as \(pK_d\) values. The PDBbind database ( http://www.pdbbind.org.cn/ ) is subdivided into two primary subsets: the general set and the refinement set . The PDBbind version 2016 dataset (named PDBbind2016) contains 9221 and 3685 unique protein-ligand complexes, while the PDBbind version 2020 dataset (named PDBbind2020) includes 14127 and 5316 protein-ligand complexes in the general and refinement sets, respectively.

Similar to the most recent state-of-the-art affinity predictors such as Pafnucy [ 20 ], DeepDTAF [ 23 ], OnionNet [ 3 ], DLSSAffinity [ 4 ], LuEtAl [ 36 ], EGNA [ 25 ] and CAPLA [ 26 ], our DAAP16 method is trained using the 9221 + 3685 = 12906 protein-ligand complexes in the general and refinement subsets of the PDBbind dataset version 2016 . Following the same training-validation set formation approach of the recent predictors such as Pafnucy, OnionNet, DeepDTAF, DLSSAffinity and CAPLA, we put 1000 randomly selected protein-ligand complexes in the validation set and the remaining 11906 distinct protein-ligand pairs in the training set. Another version of DAAP, named DAAP20, was generated using the PDBbind database version 2020 , which aligns with the training set of ResBiGAAT [ 27 ]. To avoid overlap, we filtered out protein-ligand complexes common between the PDBbind2020 training set and the six independent test sets. After this filtering process, 19027 unique protein-ligand complexes were retained for training from the initial pool of 19443 in PDBbind2020.

To ensure a rigorous and impartial assessment of the effectiveness of our proposed approach, we employ six well-established, independent blind test datasets. There is no overlap of protein-ligand complexes between the training sets and these six independent test sets.

CASF-2016.290 The 290 protein-ligand complexes, commonly referred to as CASF-2016, are selected from the PDBbind version 2016 core set ( http://www.pdbbind.org.cn/casf.php ) and have become the gold standard test set for recent affinity predictors such as DLSSAffinity [ 4 ], LuEtAl [ 36 ], EGNA [ 25 ] and CAPLA [ 26 ].

CASF-2013.87 and CASF-2013.195 Similar to the approach taken by DLSSAffinity [ 4 ], we carefully curated 87 unique protein-ligand complexes from the CASF-2013 dataset, which originally consists of 195 complexes ( http://www.pdbbind.org.cn/casf.php ). These 87 complexes were chosen to ensure no overlap with our training set or the CASF-2016 test set. Additionally, we use the entire set of 195 complexes as another test set, named CASF \(-\) 2013.195.

ADS.74 This test set from SFCNN [ 24 ] comprises 74 protein-ligand complexes sourced from the Astex diverse set [ 37 ].

CSAR-HiQ.51 and CSAR-HiQ.36 These two test datasets contain 51 and 36 protein-ligand complexes from the well-known CSAR [ 38 ] dataset. Recent affinity predictors such as EGNA [ 25 ], CAPLA and ResBiGAAT [ 26 , 27 ] have employed CSAR as a benchmark dataset. To get our two test datasets, we have followed the procedure of CAPLA and filtered out protein-ligand complexes with duplicate PDB IDs from two distinct CSAR subsets containing 176 and 167 protein-ligand complexes, respectively.

Input features

Given protein-ligand complexes in the datasets, we extract three distinctive features from proteins, ligands, and protein-ligand binding pockets. We describe these below.

Protein representation

We employ three distinct features for encoding protein sequences: one-hot encoding of amino acids, a Hidden Markov model based on multiple sequence alignment features (HHM), and seven physicochemical properties.

In the one-hot encoding scheme for the 20 standard amino acids and non-standard amino acids, each amino acid is represented by a 21-dimensional vector. This vector contains twenty “0 s” and one “1”, where the position of the “1” corresponds to the amino acid index in the protein sequence.

To construct the HHM features, we have run an iterative searching tool named HHblits [ 39 ] against the Uniclust30 database ( http://wwwuser.gwdg.de/~compbiol/uniclust/2020_06/ ) as of June 2020. This process allows us to generate HHM sequence profile features for the proteins in our analysis. Each resulting .hhm feature file contains 30 columns corresponding to various parameters such as emission frequencies, transition frequencies, and Multiple Sequence Alignment (MSA) diversities for each residue. Like EGNA, for columns 1 to 27, the numbers are transformed into frequencies using the formula \(f = 2^{-0.001*p}\) , where f represents the frequency, and p is the pseudo-count. This transformation allows the conversion of these parameters into frequency values. Columns 28 to 30 are normalized using the equation: \(f = \frac{0.001*p}{20}\) . This normalization process ensures that these columns are appropriately scaled for further analysis and interpretation.

The seven physicochemical properties [ 14 , 29 ] for each amino acid residue are steric parameter (graph shape index), hydrophobicity, volume, polarisability, isoelectric point, helix probability, and sheet probability. When extracting these three features for protein residues, we focused exclusively on the 20 standard amino acid residues. If a residue is non-standard, we assigned a feature value of 0.0.

In our approach, we initially concatenate all three features sequentially for the entire protein sequence. Subsequently, to enhance the specificity of our model, we employ a filtering strategy where residues lacking donor [ 40 ], hydrophobic [ 31 ], and \(\pi \) -stacking [ 32 ] atoms within their amino acid side chains are excluded from the analysis. Additionally, to prevent overlap, we select unique residues after identification based on donor, hydrophobic, or \(\pi \) -stacking atoms for each protein sequence. The rationale behind this filtering is to focus on residues that are actively involved in critical interactions relevant to protein-ligand binding. The resulting feature dimension for each retained protein residue is 58. This feature set includes one-hot encoding of amino acids, a Hidden Markov model based on multiple sequence alignment features (HHM), and seven physicochemical properties. These features are comprehensively summarised in Table  9 for clarity.

Considering the variable numbers of residues that proteins can possess, we have considered a standardized protein sequence length to align with the fixed-size requirements of deep learning algorithms. In our initial experiments exploring various sequence lengths in the datasets, we found that a maximum length of 500 yields better performance in terms of pearson correlation coefficient (R) and mean absolute error (MAE). If the number of selected residues falls below 500, we pad the sequence with zeros; conversely, if it exceeds 500, we truncate it to 500 from the initial position of the sequence. The final dimension of each protein is \(500\times 58\) .

Ligand representation

We use SMILES to represent ligands. SMILES is a widely adopted one-dimensional representation of chemical structures of ligands [ 41 ]. To convert ligand properties such as atoms, bonds, and rings from ligand SDF files into SMILES strings, we use the Open Babel chemical tool [ 42 ]. The SMILES strings comprise 64 unique characters, each corresponding to a specific numeric digit ranging from 1 to 64. For example, the SMILES string “HC(O=)N” is represented as [12, 42, 1, 48, 40, 31, 14]. In line with our protein representation approach, we set a fixed length of 150 characters for each SMILES string.

Various distance measures that potentially capture protein-ligand interactions. In the figure, \(d_{ij}\) represents the distance between a donor (D), hydrophobic (H), or \(\pi \) -stacking (S) atom i in the protein and the corresponding acceptor (A), hydrophobic (H), or \(\pi \) -stacking (S) atom j in the ligand. Empty circles represent other atom types. Different colour lines represent different types of interactions

Binding pocket representation

A binding pocket refers to a cavity located either on the surface or within the interior of a protein. A binding pocket possesses specific characteristics that make it suitable for binding a ligand [ 43 ]. Protein residues within the binding pocket region exert a direct influence, while residues outside this binding site can also have a far-reaching impact on affinity prediction. Among various protein-ligand interactions within the binding pocket regions, donor-acceptor atoms [ 30 ], hydrophobic contacts [ 31 , 32 ], and \(\pi \) -stacking [ 31 , 32 ] interactions are the most prevalent, and these interactions could significantly contribute to the enhancement of affinity score prediction. The formation of the protein-ligand complexes involves donor atoms from the proteins and acceptor atoms from the ligands. This process is subject to stringent chemical and geometric constraints associated with protein donor groups and ligand acceptors [ 30 ]. Hydrophobic interactions stand out as the primary driving force in protein-ligand interactions, while \(\pi \) -stacking interactions, particularly involving aromatic rings, play a substantial role in protein-ligand interactions [ 32 ]. However, there are instances where donor-acceptor interactions alone may not suffice, potentially failing to capture other interactions that do not conform to traditional donor-acceptor patterns. In such scenarios, hydrophobic contacts and \(\pi \) -stacking interactions become essential as they could provide valuable insights for accurate affinity prediction.

We employ three types of distance matrices in our work shown in Fig.  5 to capture protein-ligand interactions. The first one is the donor-acceptor distance matrix , which considers distances between protein donor atoms and acceptor ligand atoms, with data sourced from mol2/SDF files. We ensure that all ligand atoms contribute to the distance matrix construction, even in cases where ligands lack explicit acceptor atoms. Furthermore, we calculate the hydrophobic distance matrix by measuring the distance between hydrophobic protein atoms and hydrophobic ligand atoms, ensuring the distance is less than \(4.5\mathring{A}\) [ 31 ]. Similarly, we compute the \(\pi \) - stacking distance matrix by considering protein and ligand \(\pi \) -stacking atoms and applying a distance threshold of \(4.0\mathring{A}\) [ 32 ]. These three types of atoms are selected from the heavy atoms, referring to any atom that is not hydrogen.

We discretize the initially calculated real-valued distance matrices representing the three types of interactions into binned distance matrices. These matrices are constrained within a maximum distance threshold of \(20\mathring{A}\) . The decision to set a maximum distance threshold of \(20\mathring{A}\) for capturing the binding pocket’s spatial context is informed by practices in both affinity prediction and protein structure prediction fields. Notably, methodologies like Pafnucy [ 20 ], DLSSAffinity [ 4 ], and EGNA [ 25 ], as well as advanced protein structure prediction models such as AlphaFold [ 28 ] and trRosetta [ 44 ], utilize a 20Å range to define interaction spaces or predict structures. This consensus on the 20Å threshold reflects its sufficiency in providing valuable spatial information necessary for accurate modeling. The distance values ranging from \(0\mathring{A} - 20\mathring{A}\) are discretized into 40 bins, each with a \(0.5\mathring{A}\) interval. Any distance exceeding \(20\mathring{A}\) is assigned to the \(41^{st}\) bin. In our experimentation, we explored different distance ranges ( \(20\mathring{A}\) , \(25\mathring{A}\) , \(30\mathring{A}\) , \(35\mathring{A}\) , and \(40\mathring{A}\) ) while maintaining a uniform bin interval of \(0.5\mathring{A}\) . Among these ranges, \(20\mathring{A}\) yielded optimal results, and as such, we adopted it for our final analysis. Following this binning process, the original real-valued distances in the matrices are substituted with their corresponding bin numbers. Subsequently, we convert the 2D distance matrix into a 1D feature vector. We concatenate the three 1D vectors representing the three distinct interactions into a single vector to construct the final feature vector. To ensure consistency, the maximum length of the feature vector is set to 1000 for each pocket.

Output representations

This binding affinity is measured in the dissociation constant ( \(K_d\) ). For simplicity in calculations, the actual affinity score \(K_d\) is commonly converted into \(pK_d\) by taking the negative logarithm of \(K_d\) .

Deep learning architectures

The proposed model architecture

We propose a deep-learning regression model to predict protein-ligand binding affinities, shown in Fig.  6 . Our model comprises three integral components: convolutional neural network (CNN), attention mechanism, and fully connected neural network (FCNN). Before feeding to the CNN block, information from three distinct feature sources (proteins, ligands, and interactions) is encoded and subsequently processed through the embedding layer. The embedding layer transforms the inputs into fixed-length vectors of a predefined size (in this case, 128 dimensions), enabling more effective feature representation with reduced dimensionality. During training, our model operates with a batch size of 16 and is optimized using the Adam optimizer and a learning rate set at 0.001. We adopt the log cosh loss function for this work to optimise the model’s performance. The training regimen consists of 200 epochs, with the best model selected based on the validation loss, and a dropout rate of 0.2 is applied. The explored hyperparameter settings are summarised in Table  10 . We have explored these settings, and after preliminary experiments, we have selected these values which are emboldened.

Convolutional neural network

Much like DLSSAffinity [ 4 ], our model employs three 1D-CNN blocks, each dedicated to processing distinct feature sources: proteins, ligands, and interactions in pockets. Each of these 1D-CNN blocks comprises three convolutional layers paired with three Maxpooling layers. The configuration of the first two 1D-CNN blocks includes 32, 64, and 128 filters, each with corresponding filter lengths of 4, 8, and 12. In contrast, the 1D-CNN block responsible for handling SMILES sequence inputs features filters with 4, 6, and 8 adjusted lengths. Each of the three 1D-CNN blocks in our model generates a 128-dimensional output. Subsequently, before progressing to the next stage, the outputs of these three 1D-CNN blocks are concatenated and condensed into a unified 384-dimensional output.

Attention mechanism

In affinity prediction, attention mechanisms serve as crucial components in neural networks, enabling models to allocate varying levels of focus to distinct facets of input data [ 5 ]. These mechanisms play a critical role in weighing the significance of different features or entities when assessing their interaction strength. The attention mechanism uses the formula below.

We use the Scaled Dot-Product Attention [ 45 ] mechanism to calculate and apply attention scores to the input data. The attention mechanism calculates query ( Q ), key ( K ), and value ( V ) matrices from the input data. In this context, Q is a vector capturing a specific aspect of the input, K represents the context or memory of the model with each key associated with a value, and V signifies the values linked to the keys. It computes attention scores using the dot product of Q and K matrices, scaled by the square root of the dimensionality ( \(d_k\) ). Subsequently, a softmax function normalises the attention scores. Finally, the output is generated as a weighted summation of the value (V) matrix, guided by the computed attention scores.

Notably, the output of the concatenation layer passes through the attention layer. The input to the attention layer originates from the output of the concatenation layer, preserving the same dimensionality as the input data. This design ensures the retention of crucial structural information throughout the attention mechanism.

Fully connected neural network

The output of the attention layer transitions into the subsequent stage within our model architecture, known as the Fully Connected Neural Network (FCNN) block. The FCNN block consists of two fully connected (FC) layers, where the two layers have 256 and 128 nodes respectively. The final stage in our proposed prediction model is the output layer, which follows the last FC layer.

Evaluation metrics

We comprehensively evaluate our affinity prediction model using five well-established performance metrics. The Pearson Correlation Coefficient (R) [ 4 , 24 , 26 , 36 ] measures the linear relationship between predicted and actual values. The Root Mean Square Error (RMSE) [ 4 , 24 , 26 ] and the Mean Absolute Error (MAE) [ 24 , 26 ] assess prediction accuracy and error dispersion. The Standard Deviation (SD) [ 4 , 24 , 26 , 36 ] evaluates prediction consistency, and the Concordance Index (CI) [ 26 , 36 ] determines the model’s ability to rank protein-ligand complexes accurately. Higher R and CI values and lower RMSE, MAE, and SD values indicate better prediction accuracy. These metrics are collectively very robust measures for comparison of our model’s performance against that of the state-of-the-art techniques in the field of affinity prediction.

N : the number of protein-ligand complexes

\(Y_{\text {act}}\) : experimentally measured actual binding affinity values for the protein-ligand complexes

\(Y_{\text {pred}}\) : the predicted binding affinity values for the given protein-ligand complexes

\(y_{\text {act}_i}\) and \(y_{\text {pred}_i}\) : respectively the actual and predicted binding affinity value of the \(i^{th}\) protein-ligand complex

a : is slope

b : interpretation of the linear regression line of the predicted and actual values. Z : the normalization constant, i.e. the number of data pairs with different label values.

h ( u ): the step function that returns 1.0, 0.5, and 0.0for \(u>0\) , \(u = 0\) , and \(u<0\) respectively.

Availability of data and materials

The program and corresponding data are publicly available on the website https://gitlab.com/mahnewton/daap .

DiMasi JA, Grabowski HG, Hansen RW (2016) Innovation in the pharmaceutical industry: new estimates of r &d costs. J Health Econ 47:20–33

Article   PubMed   Google Scholar  

Gilson MK, Zhou H-X (2007) Calculation of protein-ligand binding affinities. Ann Rev Biophys Biomol Str 36(1):21–42

Article   CAS   Google Scholar  

Zheng L, Fan J, Mu Y (2019) Onionnet: a multiple-layer intermolecular-contact-based convolutional neural network for protein-ligand binding affinity prediction. ACS Omega 4(14):15956–15965

Article   CAS   PubMed   PubMed Central   Google Scholar  

Wang H, Liu H, Ning S, Zeng C, Zhao Y (2022) Dlssaffinity: protein-ligand binding affinity prediction via a deep learning model. Phys Chem Chem Phys 24(17):10124–10133

Article   CAS   PubMed   Google Scholar  

Seo S, Choi J, Park S, Ahn J (2021) Binding affinity prediction for protein-ligand complex using deep attention mechanism based on intermolecular interactions. BMC Bioinform 22(1):1–15

Article   Google Scholar  

Deng W, Breneman C, Embrechts MJ (2004) Predicting protein- ligand binding affinities using novel geometrical descriptors and machine-learning methods. J Chem Inf Comput Sci 44(2):699–703

Li L, Wang B, Meroueh SO (2011) Support vector regression scoring of receptor-ligand complexes for rank-ordering and virtual screening of chemical libraries. J Chem Inf Modeling 51(9):2132–2138

Ballester PJ, Mitchell JB (2010) A machine learning approach to predicting protein-ligand binding affinity with applications to molecular docking. Bioinformatics 26(9):1169–1175

Li H, Peng J, Sidorov P, Leung Y, Leung K-S, Wong M-H, Lu G, Ballester PJ (2019) Classical scoring functions for docking are unable to exploit large volumes of structural and interaction data. Bioinformatics 35(20):3989–3995

Deng L, Platt J. Ensemble deep learning for speech recognition. In: Proc. Interspeech. 2014

Chen C, Seff A, Kornhauser A, Xiao J. Deepdriving: learning affordance for direct perception in autonomous driving. In: Proceedings of the IEEE International Conference on Computer Vision, 2015; pp. 2722–2730

Lin T-Y, RoyChowdhury A, Maji S (2017) Bilinear convolutional neural networks for fine-grained visual recognition. IEEE Trans Pattern Anal Mach Intell 40(6):1309–1322

Newton MH, Rahman J, Zaman R, Sattar A. Enhancing protein contact map prediction accuracy via ensembles of inter-residue distance predictors. Computational Biology and Chemistry, 2022; 107700.

Rahman J, Newton MH, Hasan MAM, Sattar A (2022) A stacked meta-ensemble for protein inter-residue distance prediction. Comput Biol Med 148:105824

Isert C, Atz K, Schneider G (2023) Structure-based drug design with geometric deep learning. Curr Opin Struct Biol 79:102548

Krentzel D, Shorte SL, Zimmer C (2023) Deep learning in image-based phenotypic drug discovery. Trend Cell Biol 33(7):538–554

Yang L, Jin C, Yang G, Bing Z, Huang L, Niu Y, Yang L (2023) Transformer-based deep learning method for optimizing admet properties of lead compounds. Phys Chem Chem Phys 25(3):2377–2385

Masters MR, Mahmoud AH, Wei Y, Lill MA (2023) Deep learning model for efficient protein-ligand docking with implicit side-chain flexibility. J Chem Inf Modeling 63(6):1695–1707

Öztürk H, Özgür A, Ozkirimli E (2018) Deepdta: deep drug-target binding affinity prediction. Bioinformatics 34(17):821–829

Stepniewska-Dziubinska MM, Zielenkiewicz P, Siedlecki P (2018) Development and evaluation of a deep learning model for protein-ligand binding affinity prediction. Bioinformatics 34(21):3666–3674

Jiménez J, Skalic M, Martinez-Rosell G, De Fabritiis G (2018) \(k_{deep}\) : protein-ligand absolute binding affinity prediction via 3d-convolutional neural networks. J Chem Inf Modeling 58(2):287–296

Li Y, Rezaei MA, Li C, Li X (2019) Deepatom: a framework for protein-ligand binding affinity prediction. In: 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 303–310, IEEE

Wang K, Zhou R, Li Y, Li M (2021) Deepdtaf: a deep learning method to predict protein-ligand binding affinity. Brief Bioinf 22(5):072

Wang Y, Wei Z, Xi L (2022) Sfcnn: a novel scoring function based on 3d convolutional neural network for accurate and stable protein-ligand affinity prediction. BMC Bioinform 23(1):1–18

Xia C, Feng S-H, Xia Y, Pan X, Shen H-B (2023) Leveraging scaffold information to predict protein-ligand binding affinity with an empirical graph neural network. Brief Bioinf. https://doi.org/10.1093/bib/bbac603

Jin Z, Wu T, Chen T, Pan D, Wang X, Xie J, Quan L, Lyu Q (2023) Capla: improved prediction of protein-ligand binding affinity by a deep learning approach based on a cross-attention mechanism. Bioinformatics 39(2):049

Abdelkader GA, Njimbouom SN, Oh T-J, Kim J-D (2023) Resbigaat: Residual bi-gru with attention for protein-ligand binding affinity prediction. Computational Biology and Chemistry, 107969

Senior AW, Evans R, Jumper J, Kirkpatrick J, Sifre L, Green T, Qin C, Žídek A, Nelson AW, Bridgland A et al (2020) Improved protein structure prediction using potentials from deep learning. Nature 577(7792):706–710

Rahman J, Newton MH, Islam MKB, Sattar A (2022) Enhancing protein inter-residue real distance prediction by scrutinising deep learning models. Sci Rep 12(1):787

Raschka S, Wolf AJ, Bemister-Buffington J, Kuhn LA (2018) Protein-ligand interfaces are polarized: discovery of a strong trend for intermolecular hydrogen bonds to favor donors on the protein side with implications for predicting and designing ligand complexes. J Computer-aided Mol Design 32:511–528

Jubb HC, Higueruelo AP, Ochoa-Montaño B, Pitt WR, Ascher DB, Blundell TL (2017) Arpeggio: a web server for calculating and visualising interatomic interactions in protein structures. J Mol Biol 429(3):365–371

Freitas RF, Schapira M (2017) A systematic analysis of atomic protein-ligand interactions in the pdb. Medchemcomm 8(10):1970–1981

Empereur-Mot C, Guillemain H, Latouche A, Zagury J-F, Viallon V, Montes M (2015) Predictiveness curves in virtual screening. J Cheminf 7(1):1–17

Li H, Zhang H, Zheng M, Luo J, Kang L, Liu X, Wang X, Jiang H (2009) An effective docking strategy for virtual screening based on multi-objective optimization algorithm. BMC Bioinf 10:1–12

Liu T, Lin Y, Wen X, Jorissen RN, Gilson MK (2007) Bindingdb: a web-accessible database of experimentally determined protein-ligand binding affinities. Nucleic Acids Res 35(suppl–1):198–201

Lu Y, Liu J, Jiang T, Guan S, Wu H. Protein-ligand binding affinity prediction based on deep learning. In: International Conference on Intelligent Computing, 2022; pp. 310–316. Springer.

Hartshorn MJ, Verdonk ML, Chessari G, Brewerton SC, Mooij WT, Mortenson PN, Murray CW (2007) Diverse, high-quality test set for the validation of protein- ligand docking performance. J Med Chem 50(4):726–741

Dunbar JB Jr, Smith RD, Yang C-Y, Ung PM-U, Lexa KW, Khazanov NA, Stuckey JA, Wang S, Carlson HA (2011) Csar benchmark exercise of 2010: selection of the protein-ligand complexes. J Chem Inf Modeling 51(9):2036–2046

Remmert M, Biegert A, Hauser A, Söding J (2012) Hhblits: lightning-fast iterative protein sequence searching by hmm-hmm alignment. Nat Methods 9(2):173–175

Hydrogen donor and acceptor atoms of the amino acid. https://www.imgt.org/IMGTeducation/Aide-memoire/_UK/aminoacids/charge/ . Accessed: 13-08-2023

Weininger D (1988) Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. J Chem Inf Comput Sci 28(1):31–36

O’Boyle NM, Banck M, James CA, Morley C, Vandermeersch T, Hutchison GR (2011) Open babel: An open chemical toolbox. J Cheminf 3(1):1–14

Google Scholar  

Stank A, Kokh DB, Fuller JC, Wade RC (2016) Protein binding pocket dynamics. Accounts Chem Res 49(5):809–815

Yang J, Anishchenko I, Park H, Peng Z, Ovchinnikov S, Baker D (2020) Improved protein structure prediction using predicted interresidue orientations. Proc Natl Acad Sci 117(3):1496–1503

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems 30

Download references

Acknowledgements

This research is partially supported by the research seed grant awarded to M.A.H.N. at the University of Newcastle. The research team acknowledges the valuable assistance of the Griffith University eResearch Service & Specialised Platforms team for granting access to their High-Performance Computing Cluster, which played a crucial role in completing this research endeavour.

This research is partially supported by the research seed Grant awarded to M.A.H.N. at the University of Newcastle.

Author information

Julia Rahman and M. A. Hakim Newton are co-first-authors and contributed equally.

Authors and Affiliations

School of Information and Communication Technology, Griffith University, 170 Kessels Rd, Nathan, 4111, QLD, Australia

Julia Rahman

Institute for Integrated and Intelligent Systems (IIIS), Griffith University, 170 Kessels Rd, Nathan, 4111, QLD, Australia

M. A. Hakim Newton & Abdul Sattar

School of Information and Physical Sciences, University of Newcastle, University Dr, Callaghan, 2308, NSW, Australia

M. A. Hakim Newton

Department of Computer Science & Engineering, Bangladesh University of Engineering and Technology, Palashi, 1205, Dhaka, Bangladesh

Mohammed Eunus Ali

You can also search for this author in PubMed   Google Scholar

Contributions

The contributions of the authors to this work were as follows: J.R. and M.A.H.N. played equal roles in all aspects of the project, including conceptualization, data curation, formal analysis, methodology, software development, and writing of the initial draft. M.E.A. helped in the concept development, review and editing of the manuscript. A.S. actively engaged in discussions, facilitated funding acquisition, provided supervision, and thoroughly reviewed the manuscript.

Corresponding author

Correspondence to Julia Rahman .

Ethics declarations

Competing interests.

No Conflict of interest is declared.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Rahman, J., Newton, M.A.H., Ali, M.E. et al. Distance plus attention for binding affinity prediction. J Cheminform 16 , 52 (2024). https://doi.org/10.1186/s13321-024-00844-x

Download citation

Received : 01 December 2023

Accepted : 24 April 2024

Published : 12 May 2024

DOI : https://doi.org/10.1186/s13321-024-00844-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Binding affinity
• Distance matrix
• Donor-acceptor
• Hydrophobicity
• \(\pi \) -Stacking
• Deep learning

Journal of Cheminformatics

ISSN: 1758-2946

• Submission enquiries: [email protected]

Destiny Unbound: A Look at How Far from Home Students Go to College

• Published: 10 May 2024

Cite this article

• Robert K. Toutkoushian   ORCID: orcid.org/0000-0001-7570-8546 1 ,
• Stephen Mayfield 2 &
• Samantha M. R. Jelks 3  

34 Accesses

Explore all metrics

One issue that has received little attention is how students factor distance from home into their decisions about college. In this study, we used data from the Education Longitudinal Survey of 2002 (ELS:02) to examine the distances between a student’s home and the colleges to which they applied, and how far from home they enrolled. We focused on how demand- and supply-side factors were related to the distances applied and enrolled. We tested the sensitivity of our findings to alternative ways of measuring the supply of postsecondary education within commuting distance, and identified factors associated with differences between a student’s application and enrollment distances. Finally, we used quantile regression analysis to determine if the associations between demand- and supply-side factors and distances applied and enrolled varied along the distance distributions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA) Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Geographical Constraints and College Decisions: How Does For-Profit College Play in Student’s Choice?

Geographic Inequalities and Access to Higher Education: Is the Proximity to Higher Education Institution Associated with the Probability of Attendance in England?

Does living closer to a university increase educational attainment a longitudinal study of aspirations, university entry, and elite university enrolment of australian youth, data availability.

The data used in this study can be obtained through a license with the National Center for Education Statistics.

Although it is a more recent survey, the High School Longitudinal Study of 2009 (HSLS:09) was not used because it only provides information on up to three institutions to which a student applied, and does not contain the student’s ZIP code during high school.

Applications and enrollments from students from the mainland to colleges in Hawaii and Alaska were included.

All sample sizes are rounded per NCES requirements. The data used in all tables have been weighted using the ELS variable F2F1WT to account for the complex sampling design used in the survey.

The student ZIP code was collected in the 2004 ELS first Follow-up Survey. We also used this ZIP code to infer the students’ county, which is in turn aggregated up to the commuting zone level. The ZIP codes for colleges were gathered from the 2004 IPEDS file. If no college ZIP code existed in IPEDS 2004, we used the ZIP codes as reported in IPEDS surveys for other years.

The straight-line distances were derived using Stata’s Geodist command. We acknowledge that there are other ways to measure distance, including the commuting distance between two points in miles and time. The computation of these alternate measures would require connecting the computer with the restricted-use data to the internet, which is prohibited by the licensing agreement with IES.

The clusters of counties for each commuting zone were created by the USDA Economic Research Service using “journey to work” data from the 2000 US Census.

Median family income was generated as a weighted average of the counties that comprise each commuting zone (i.e., median household income times the number of households in county). The unemployment rate per commuting zone was an average for the years 2004–2006.

We also estimated all of the models using standard errors clustered at the state level and found no appreciable differences in the significance levels for the variables.

While ordinary least squares selects coefficients that minimize the sum of squared deviations, quantile regression selects coefficients that minimize the sum of weighted absolute deviations. The weights in quantile regression analysis allow separate parameters to be estimated for different percentiles of the distribution. More details on quantile regression analysis can be found in Koenker and Hallock ( 2001 ).

Due to space limitations, the results for the state variables in the models that follow are not shown but are available upon request.

The results were fairly robust when we used either the median or maximum distances applied.

Alm, J., & Winters, J. (2009). Distance and intrastate college student migration. Economics of Education Review, 28 , 728–738.

Article   Google Scholar  

Baryla, E., & Dotterweich, D. (2001). Student migration: Do significant factors vary by region? Education Economics, 9 , 269–280.

Becker, W., & Toutkoushian, R. (2013). On the meaning of markets in higher education. In M. Paulsen (Ed.), Higher education: Handbook of theory and research (Vol. XXVIII, pp. 323–376). Springer.

Chapter   Google Scholar  

Bourdieu, P. (1973). Cultural reproduction and social reproduction. In R. Brown (Ed.), Knowledge, education, and cultural change. Routledge.

Google Scholar  

Bourdieu, P. (1977). Outline of a theory of practice . Cambridge University Press.

Book   Google Scholar  

Bourdieu, P. (1990). The logic of practice . Stanford University Press.

Byun, S., Irvin, M., & Meece, J. (2015). Rural–nonrural differences in college attendance patterns. Peabody Journal of Education, 90 , 263–279.

Coleman, J. (1988). Social capital in the creation of human capital. American Journal of Sociology, 94 , 95–120.

Cooke, T., & Boyle, P. (2011). The migration of high school graduates to college. Educational Evaluation and Policy Analysis, 33 , 202–213.

Dache-Gerbino, A. (2018). College desert and oasis: A critical geographic analysis of local college access. Journal of Diversity in Higher Education, 11 , 97–116.

De Graaf, N., De Graaf, P., & Kraaykamp, G. (2000). Parental cultural capital and educational attainment in the Netherlands: A refinement of the cultural capital perspective. Sociology of Education, 73 , 92–111.

Denzler, S., & Wolter, S. (2011). Too far to go? Does distance determine study choices? IZA Discussion Paper No. 571. The Institute for the Study of Labor.

DesJardins, S., Ahlburg, D., & McCall, B. (2006). An integrated model of application, admission, enrollment, and financial aid. Journal of Higher Education, 77 , 381–429.

Dill, D. (1997). Higher education markets and public policy. Higher Education Policy, 10 , 167–185.

Dillon, E., & Smith, J. (2017). Determinants of the match between student ability and college quality. Journal of Labor Economics, 35 , 45–66.

Do, C. (2004). The effects of local colleges on the quality of college attended. Economics of Education Review, 23 , 249–257.

Eagan, K., Stolzenberg, E., Ramirez, J., Aragon, M., Suchard, M., & Rios-Aguilar, C. (2016). The American freshman: Fifty-year trends 1966–2015 . Higher Education Research Institute.

Evans, M., Kelley, J., Sikora, J., & Treiman, D. J. (2010). Family scholarly culture and educational success: Books and schooling in 27 nations. Research in Social Stratification and Mobility, 28 , 171–197.

Frenette, M. (2004). Access to college and university: Does distance to school matter? Canadian Public Policy, 30 , 427–443.

Frenette, M. (2006). Too far to go on? Distance to school and university participation. Education Economics, 14 , 31–58.

Garza, A., & Fullerton, A. (2018). Staying close or going away: How distance to college impacts the educational attainment and academic performance of first-generation college students. Sociological Perspectives, 61 , 164–185.

Ghatak, S., Levine, P., & Price, S. (1996). Migration theories and evidence: An assessment. Journal of Economic Surveys, 10 , 159–198.

Gibbons, S., & Vignoles, A. (2012). Geography, choice and participation in higher education in England. Regional Science and Urban Economics, 42 , 98–113.

González Canché, M. (2018). Nearby college enrollment and geographical skills mismatch: (Re)conceptualizing student out-migration in the American higher education system. The Journal of Higher Education, 89 , 892–934.

Greenwood, M. (1975). Research on internal migration in the United States: A survey. Journal of Economic Literature, 13 , 397–433.

Griffith, A., & Rothstein, D. (2009). Can’t get there from here: The decision to apply to a selective college. Economics of Education Review, 28 , 620–628.

Guillory, J. (2023). Cultural capital: The problem of literary canon formation . University of Chicago Press.

Han, P. (2014). A literature review on college choice and marketing strategies for recruitment. Family and Consumer Sciences Research Journal, 43 , 120–130.

Hill, L., Bregman, A., & Andrade, F. (2015). Social capital for college: Network composition and access to selective institutions among urban high school students. Urban Education, 50 , 316–345.

Hillman, N. (2016). Geography of college opportunity: The case of education deserts. American Educational Research Journal, 53 , 987–1021.

Hirschl, N., & Smith, C. (2020). Well-placed: The geography of opportunity and high school effects on college attendance. Research in Higher Education, 61 , 567–587.

Holloman, D., & Nolen, A. (2008). Recruiting place bound students: The influence of location on the college choice of high-school seniors. Metropolitan Universities, 19 , 41–53.

Hossler, D., & Gallagher, K. (1987). Studying student college choice: A three-phase model and the implications for policymakers. College and University, 62 , 207–221.

Hossler, D., Schmit, J., & Vesper, N. (1999). Going to college: How social, economic, and educational factors influence the decisions students make . The Johns Hopkins University Press.

Ishitani, T. (2011). The determinants of out-migration among in-state college students in the United States. Research in Higher Education, 52 , 107–122.

Jaeger, M., & Breen, R. (2016). A dynamic model of cultural reproduction. American Journal of Sociology, 121 , 1079–1115.

Kim, D. (2004). The effect of financial aid on students’ college choice: Differences by racial groups. Research in Higher Education, 45 , 43–70.

Klasik, D., Blagg, K., & Pekor, Z. (2018). Out of the education desert: How limited local college options are associated with inequity in postsecondary opportunities. Social Sciences, 7 , 165–190.

Koenker, R., & Hallock, K. (2001). Quantile regression. Journal of Economic Perspectives, 15 , 143–156.

Leppel, K. (1993). Logit estimation of a gravity model of the college enrollment decision. Research in Higher Education, 34 , 387–398.

Lillis, M., & Tian, R. (2008). The impact of cost on college choice: Beyond the means of the economically disadvantaged. Journal of College Admission, 200 , 4–14.

Marginson, S. (2019). Limitations of human capital theory. Studies in Higher Education, 44 , 287–301.

Mattern, K., & Wyatt, J. (2009). Student choice of college: How far do students go for an education? Journal of College Admission, 203 , 18–29.

McHugh, R., & Morgan, J. (1984). The determinants of interstate student migration: A place-to-place analysis. Economics of Education Review, 3 , 269–278.

Mishra, S. (2020). Social networks, social capital, social support and academic success in higher education: A systematic review with a special focus on ‘underrepresented’ students. Educational Research Review, 29 , 1–24.

Mixon, F. (1992). Factors affecting college student migration across states. International Journal of Manpower, 13 , 25–32.

Mixon, F., & Hsing, Y. (1994). The determinants of out-of-state enrollment in higher education: A tobit analysis. Economics of Education Review, 13 , 329–335.

Mortenson, T. (1996). Freshmen enrolling in college farther from home, but who can afford to go so far away? Postsecondary Education Opportunity, 50 , 8–13.

Mortenson, T. (1998). Interstate migration of college undergraduates. Postsecondary Education Opportunity, 70 , 1–12.

Morton, T., Ramirez, N., Meece, J., Demetriou, C., & Panter, A. (2018). Perceived barriers, anxieties, and fears in prospective college students from rural high schools. High School Journal, 101 , 155–176.

Niu, S. (2015). Leaving home state for college: Differences by race/ethnicity and parental education. Research in Higher Education, 56 , 325–359.

Perna, L. (2006). Studying college access and choice: A proposed conceptual model. In J. Smart (Ed.), Higher education: Handbook of theory and research. (Vol. XXI). Kluwer Academic Publishers.

Pryor, J., Hurtado, S., Saenz, V., Lindholm, J., Korn, W., & Mahoney, K. (2006). The American freshman: National norms for Fall 2005 . Higher Education Research Institute. Retrieved July 9, 2013 from http://heri.ucla.edu/PDFs/ResearchBrief05.PDF

Sandefur, G., Meier, A., & Campbell, M. (2006). Family resources, social capital, and college attendance. Social Science Research, 35 , 525–553.

Spiess, C., & Wrohlich, K. (2010). Does distance determine who attends a university in Germany? Economics of Education Review, 29 , 470–479.

Stephenson, A., Heckert, A., & Yerger, D. (2016). College choice and the university brand: Exploring the consumer decision framework. Higher Education, 71 , 489–503.

Tinto, V. (1973). College proximity and rates of college attendance. American Educational Research Journal, 10 , 277–293.

Toutkoushian, R. (2001a). Do parental income and educational attainment affect the initial choices of New Hampshire’s college-bound students? Economics of Education Review, 20 , 245–262.

Toutkoushian, R. (2001b). Changes in the demand for public land-grant universities: The use of panel datasets. Research in Higher Education, 42 , 679–707.

Toutkoushian, R., & Hillman, N. (2012). The impact of state appropriations and grants on access to higher education and state outmigration. The Review of Higher Education, 36 , 51–90.

Toutkoushian, R., & Paulsen, M. (2016). Economics of higher education: Background, concepts, and applications . Springer.

Tramonte, L., & Willms, J. D. (2010). Cultural capital and its effects on education outcomes. Economics of Education Review, 29 , 200–213.

Tuckman, H. (1970). Determinants of college student migration. Southern Economic Journal, 37 , 184–189.

Turley, R. (2009). College proximity: Mapping access to opportunity. Sociology of Education, 82 , 126–146.

Turner, S. (2017). Education markets: Forward-looking policy options . Hutchins Center Working Paper #27. Brookings.

Wells, R., Manly, C., Kommers, S., & Kimball, E. (2019). Narrowed gaps and persistent challenges: Examining rural–nonrural disparities in postsecondary outcomes over time. American Journal of Education, 126 , 1–31.

Zhang, L., & Ness, E. (2010). Does state merit-based aid STEM brain drain? Educational Evaluation and Policy Analysis, 32 , 143–165.

Download references

Acknowledgements

We are grateful to the Spencer Foundation for financial support for this project (grant #201900227). An earlier version of this paper was presented at the annual meeting of the Southern Economic Association. We would like to thank participants at the Southern Economic Association conference and Steve DesJardins for comments on an earlier draft of this study.

Author information

Authors and affiliations.

Institute of Higher Education, University of Georgia, Meigs Hall 114, Athens, GA, 30602, USA

Robert K. Toutkoushian

RTI International, Durham, NC, USA

Stephen Mayfield

Institutional Research & Decision Support, Ellucian, Strategy & Planning, Managed Services, Atlanta, GA, USA

Samantha M. R. Jelks

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Robert K. Toutkoushian .

Ethics declarations

Conflict of interest.

I have no known conflict of interest to disclose.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Toutkoushian, R.K., Mayfield, S. & Jelks, S.M.R. Destiny Unbound: A Look at How Far from Home Students Go to College. Res High Educ (2024). https://doi.org/10.1007/s11162-024-09790-x

Download citation

Received : 24 June 2023

Accepted : 02 April 2024

Published : 10 May 2024

DOI : https://doi.org/10.1007/s11162-024-09790-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• College choice
• Find a journal
• Publish with us
• Track your research