app development research paper

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Mobile app development in health research: pitfalls and solutions

Aaron j. siegler.

1 Department of Behavioral Sciences and Health Education, Emory University, Atlanta, GA, USA;

Justin Knox

2 Department of Epidemiology, Columbia University, New York, NY, USA;

José A. Bauermeister

3 Department of Family & Community Health, University of Pennsylvania, Philadelphia, PA, USA;

Jesse Golinkoff

4 Department of Family & Community Health, University of Pennsylvania, Philadelphia, PA, USA;

Lisa Hightow-Weidman

5 Department of Medicine, University of North Carolina, Chapel Hill, NC, USA;

Hyman Scott

6 Bridge HIV, San Francisco Department of Public Health, San Francisco, CA, USA

Mobile app health research presents myriad opportunities to improve health, and simultaneously introduces a new set of challenges that are non-intuitive and extend beyond typical training received by researchers. Informed by our experiences with app development for health research, we discuss some of the most salient pitfalls when working with emerging technology as well as potential strategies to avoid or resolve these challenges. To address challenges at the project level, we suggest strategies that researchers can use to future-proof their research, such as using theory and involving those with app development expertise as part of a research team. At the structural level, we include a new model to characterize the relationship between technology- and research-timelines, and provide ideas regarding how to best address this challenge. Given that screen-based time now predominates our lived experiences, it is important that health researchers have the capacity and structural support to develop interventions that utilize these technologies, assess them rigorously, and ensure their timely and equitable dissemination.

Introduction

The use of mobile applications (apps) is pervasive across every aspect of daily life. Smartphones have been adopted faster than nearly any other technological innovation in history, such that it is now nearly universal in the United States (81% of all US adults, including 96% of adults ages 18–29) ( 1 - 3 ). This phenomenon has also occurred across the globe, with over 3.3 billion people using smartphones worldwide ( 4 ). Adults spend 2.5–5 hours per day on their phones, or 13–16% of their waking hours. This trend is even more pronounced among US adolescents, 95% of whom either own or have access to a smartphone, with nearly half reporting being online on a near-constant basis ( 5 ). Despite some health concerns, particularly related to excessive screen time, there is growing interest in leveraging smart phone technologies to promote health through mobile apps ( 6 - 8 ).

The development and use of mobile health apps is rapidly increasing, with a wide variety of functions such as self-monitoring of chronic health conditions, medication adherence reminders, and direct interactions with the health care system ( 9 - 12 ). Mobile health apps are increasingly being used even in situations when clinical care is provided in-person, as they can be used to help tailor patient-provider communication and support patient self-management and care engagement ( 9 ). Multiple systematic reviews, usually grouped by health condition or sub-population, have summarized the growing evidence base for the effectiveness of mobile health apps ( 10 , 11 ). For example, a recent meta-analysis of digital interventions that address alcohol consumption in community-dwelling populations, including many mobile health apps, found moderate-quality evidence that digital interventions decrease unhealthy alcohol consumption ( 13 ). A recent systematic review of the use of mobile health apps for substance use disorders concluded that the heterogeneity of mobile health apps made reaching a consensus about their overall effectiveness challenging ( 14 ). This review also noted that mobile health apps should fully capitalize on the technology’s capacity to tailor itself to meet the individual needs of users ( 14 ). Achieving this, however, will likely require a better understanding of how people incorporate technology into their everyday lives, as well as research into effective ways to disseminate efficacious interventions into more diverse clinical and community settings. Another review noted the great promise for mobile health apps to make an impact in low- and middle-income countries, where access to medical care is often limited but smartphone ownership is widespread ( 12 ).

A growing number of health research programs investigate app-based interventions, yet many research teams conduct their app research in isolation. There are substantial benefits to be gained by sharing knowledge across disciplines, particularly in navigating common challenges and leveraging areas of strength. In this article, we discuss opportunities and pitfalls for mobile health app research, and propose solutions to facilitate success and overcome challenges.

Opportunities

Leveraging device, operating system, and potential for scale.

Mobile apps are able to leverage the strengths of host device hardware and operating systems. A review by Harari and colleagues has documented the numerous sensors and data collected by research apps ( 15 ), including accelerometer (coordinates, duration of movement), GPS scan (geolocation), clock (time), light sensor (ambient light), and microphone (audio). These data sources can be used creatively to develop interventions. For instance, a sun protection trial combined GPS data with real-time forecasts and time of day information to provide guidance regarding risk of sunburn and time until needed reapplication of sunscreen ( 16 ). A number of physical activity trials have used smartphone accelerometer data to collect physical activity data, and display such data as part of the intervention to study participants ( 17 ). Operating systems also provide rich interactive and monitoring features, including alarms, notifications, call logs, text logs, and system usage information. Data generated by these features can be highly useful; one study validated an algorithm to predict the total amount of users’ sleep based on their smartphone screen being on or off, with an average error of only 7% ( 18 ). Alarms/notifications are a main feature of many app-based interventions, providing a way to communicate updated data-informed progress towards goals, motivational messages, and re-engagement messages ( 17 , 19 - 21 ). Another important benefit of the flexibility of app systems is their capacity to interface with a wide variety of other devices such as pedometers ( 17 ) and pill bottle cap sensors ( 22 ).

The ability to positively influence health at a large scale is an intriguing advantage of successful mobile apps. Traditionally, the gold standard for impactful individual-level behavioral interventions has been evidence-based, multi-session interventions that are delivered in person. This approach provides a high level of exposure to a potentially tailored intervention, although it comes at a high cost by requiring substantial staff time and materials for each person newly engaged. Such multi-session interventions are typically sequentially planned to control the order in which a participant is exposed to an intervention, potentially enhancing the intervention’s effect, but also challenging the logistics of delivery. Mobile health apps have the potential to provide users with a high level of exposure (smartphones are ubiquitous and heavily used), while only requiring staff time that is fixed to the development and monitoring of the sequentially-designed app intervention, with low additional cost per person reached. Other technology-based intervention modalities, such as text messaging and website-based interventions, may have similar benefits of scale.

Tailoring and measurement

Apps are a natural fit for providing tailored health information, with the potential to build in automated tailoring by user groups or by disease condition. Apps are created for a broad array of conditions that require tailored materials, ranging from tele-rehabilitation for people with multiple sclerosis ( 23 ) to interventions to address childhood obesity ( 24 ). App platforms allow for each user group to receive an intervention appropriate for and customized to their experiences. This has the additional benefit of facilitating more successful inclusion of groups experiencing health disparities such as youth, sexual, and racial minorities ( 25 , 26 ). Through tailoring, mobile apps have the potential to engage persons in their health promotion in new and innovate ways, which are moreover less dependent on existing healthcare structures. For instance, apps can help users collect and track data on a particular health behavior and can return information tailored to that individual, such as their stage in transtheoretical model ( 27 ), thereby optimally facilitating behavioral change.

Paradata, automated process data collected as users interact with a smartphone app, is an important additional tool to gain insight on how users engage with an app ( 28 , 29 ). Examples of paradata include log-in/log-out times; time spent in the app overall and by each app feature; and number of clicks through each app feature. Used in combination with the primary research outcomes data, paradata provides insight into user preferences and app use patterns. It can help app researchers to understand why an app feature may or may not have met expected outcomes, and provide direction for further tailoring or updates.

Technology versus research timelines

Technology product development is characterized by a period of ascent characterized by high innovation followed by a phase of maturity, and then a period of decline ( 30 ). There can be a mismatch between the timelines of NIH-funded research and technological products. NIH research grant cycles predominantly adhere to a 10-year timeline, yet technology cycles may occur more rapidly ( Figure 1 ). To elaborate: a typical NIH-funded research cycle begins with idea generation, seed funding and baseline data collection, and first grant application (2 years), and then proceeds to a planning grant (e.g., R34/R21) and its implementation (3 years), clinical trial phase (5 years), and results dissemination (1 year), totaling a more than 10-year cycle. It is worth noting that some of these research phases may be skipped, such as if sufficient baseline data exist to avoid the seed funding period. Yet other issues may extend research timelines, such as grant resubmissions. In contrast to the linear and timeline-based process of research, there is greater variability in technological uptake, which tends to ebb and flow based on development of technological breakthroughs such as new platforms and interfaces. These types of breakthroughs may challenge the 10-year NIH cycle. For instance, if an innovative product such as the Nintendo Wii or apps such as Pokemon Go entice researchers and they initiate the NIH-funded research cycle during ascent or maturity periods, it is possible for product sales to be (I) obsolete, ended prior to research completion, (II) simultaneous, ending at the same time as the research, or (III) continuing, with sales ongoing or even growing as the research ends. In the first two scenarios, the product platform has ended before research regarding a particular intervention can be brought to scale.

Technology versus research cycles.

Given rapid changes in technology, it is difficult to predict when disruptive technology may break through and interrupt research plans, including those regarding app development or configuration. More subtle than disruptive innovations, but perhaps equally important, are changing norms in the design of technology interfaces. Design and user interface norms shift rapidly, with older fonts, icons, graphics, and functions quick to appear dated. Without updating, older interfaces may appear stale or nonintuitive, potentially impacting user willingness to engage with the technology. This is especially likely to impact apps grounded in gamification and graphics features, which are likely to require more frequent updating.

App development

There is an inherent information asymmetry between professional app developers’ expertise in coding and researchers possessing terminal degrees in unrelated fields. Information asymmetry is classically described as an imbalance of power that can lead to poor outcomes. Researchers with projects that involve app development either work with an app development agency, work with a free-lance app developer, or hire a full-time employee with app development experience. Due to limited experience and information, researchers are likely to have difficulty choosing the optimal approach. Moreover, any final contract may be insufficient to facilitate successful development required by the researcher ( 31 ). Once app development has begun, most researchers do not have the requisite information or experience to properly oversee the process. For example, an app developer may not be creating code of a sufficiently high quality, yet a researcher may not be able to detect this until the contract is finished and problems emerge with the app’s functioning. Moreover, researchers without app development experience may miss key areas during its design that can affect both users’ experience and interface with the app. Key areas of development include such things as coding architecture, common features (e.g., login with new technology such as fingerprint or face recognition), navigating institutional (e.g., university) and professional [e.g., Health Information Portability and Accountability Act (HIPAA)] security standards, and clarifying data export requirements. When challenges arise, any unplanned but desired app features are likely to result in substantial shifts in timelines and additions to development costs beyond the initially agreed fee.

Researchers should be particularly clear about the data export functionality of their app so that they can ensure data are being collected and exported in a way that facilitates its analysis. This area requires particular attention because it is foreign to most developers who usually focus on business-based outputs of ad clicks and sales. Even those developers accustomed to working with academics may struggle, especially as developers often code in teams, and research expertise may not be uniform across a technology team.

Technical challenges can be exacerbated by the high expenses required to create apps. Apps sufficiently nuanced to fulfill the needs of researchers, and that meet current HIPAA and other security standards, can be costly. The costs of app development range broadly, but even for a relatively low-feature health app, researchers should plan for costs over $150,000 US dollars if building from scratch (building from an existing platform can substantially change cost), a cost that may not fit into the budgets of NIH funding mechanisms typically used to fund pilot studies ( 32 ). An important component contributing to high costs are requisite security features for apps dealing with health information. Security concerns are not hypothetical; for example, in 2015, the health information of over 100 million individuals was breached ( 33 ). In other years between 2013 and 2017, more than 10 million individuals per year were impacted by health information breaches ( 33 ).

Translation and download problems

Once developed and tested in a clinical trial, apps demonstrated to be efficacious should be disseminated. Yet pathways to do so are relatively uncharted, and face two substantial problems. The translation problem is that health apps developed by researchers may never be translated into production models that can be downloaded by the public (and not just solely accessed by research subjects). In a search of NIH RePORTER conducted in late 2018, we identified 40 grants awarded to develop mobile apps relating to HIV, with 17 of these grants having completed their years of award. These grants represented a diverse portfolio of research, with target outcomes ranging from prevention (e.g., condom use) to treatment adherence, and target groups including general populations, racial/ethnic minorities, injection drugs users, men who have sex with men, and cis-gender women—indicating that any problems with performance were likely attributable to challenges spanning research domains and target populations. We performed a review of information for the 40 grants, both within RePORTER and within publications citing relevant grant numbers, to identify information such as a name or keyword that would allow us to search for resulting apps. We then searched the Google Play and iTunes App stores. We were unable to locate any apps from HIV grants in RePORTER that could be used by members of the public. We identified 2 apps that could be downloaded and used only by research participants.

The download problem is that even once made public, persons most in need of services are unlikely to download evidence-based apps unless they (I) are aware of the app, (II) believe the app provides substantial utility, and (III) believe it is better than existing apps. The app market is already inundated with non-evidence-based health apps. For instance, a review of mobile apps for HIV prevention identified 285 publicly available apps, but most (71%) were not developed by academic or public health entities, and none dealt with a key component of current prevention efforts: HIV pre-exposure prophylaxis ( 34 ). Similarly, a review of apps promulgated to support mental health found that only 10% offered support that was consistent with principles of evidence-based practice ( 35 ).

Sustainability

In addition to translation and download problems, apps that are disseminated require continual resources for updating and maintenance. Without this, apps quickly become dated and can stop functioning. This makes a post-research translation even more challenging. If not developed using a profit-seeking model, research funds are finite and tied to a specific set of proposed activities, putting most of this work beyond the scope of the vast majority of research proposals.

Conducting research informed by theory that applies rigorous methodologies

Research grounded in behavioral theory that uses rigorous and appropriate methodologies can produce findings that are generalizable beyond the life of the technology used in the research. There are multiple examples of this in research using the now defunct personal digital assistant (PDA). For instance, a recent PDA-based study used ecological momentary assessments (EMA), a method that involves repeated sampling of subjects experiences in their natural environments in real time, and found that side-effects and self-management among cancer survivors offer opportunities for tailored care programming ( 36 ). Another study used PDA and EMA, finding evidence that supports emotional regulation being measured as a trait ( 37 ). A study using the Technology Acceptance Model found that 71% of the variation of physicians intention to use a new device was explained by domains of perceived usefulness and perceived ease of use ( 38 ). Conversely, failure to base interventions on theory or functionality that can extend beyond a single technology leads to conclusions that are outdated by their time of publication. For example, the main conclusion of a 2013 article regarding PDAs (when such devices were off-market) was that PDA use should be scaled up among nursing students. This article could have benefitted from considering the benefits of incorporating changing technology into nursing education, in general, as well as the challenges with the changing pace of such technological solutions ( 39 ).

Leverage screen capture technology to document intervention

Technology change not only produces challenges for research, but also opportunities. Journals produce electronic supplements to publications, and those dedicated to protocol publication are an optimal venue for documentation of app-based interventions. Screenshots and verbal descriptions are useful to indicate functionality of websites, and are commonly used ( 40 ). The chief limitation of this approach is that it is challenging to describe interactive platforms with static images and text. We propose an alternative: to create screen recordings and voiceovers to provide a “walk-through” for each primary function of a study app. It is likely that such videos would be a valuable resource for future researchers and developers to adapt successful interventions for new platforms. Walk-throughs can be created, at no cost other than a limited amount of staff time, by using native screen recording functionality of a mobile phone. To our knowledge, this approach has not previously been used to document app-based interventions. For proprietary intervention components, video materials could be released alongside publication of trial results to facilitate future dissemination.

Short-circuit timelines and facilitate dissemination

To improve the responsiveness of the NIH funding cycle in order to more effectively conduct research on or using technology, a number of steps could be taken. One option is for researchers to consider alternative grant mechanisms within NIH, such as the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs that are designed to support health products produced by small businesses in collaboration with research partners. The 2.5-year total research timeline for SBIR/STTR (a 6-month pilot, phase I and 2-year trial, phase II) allows it to be much more responsive to changes in technology, with a targeted goal of commercializing a product. Another option is to leverage NIH center grants and other mechanisms, such as U-level trials networks, to bring promising interventions to scale more quickly. For instance, the UNC/Emory Center for Innovative Technology (iTech), part of the Adolescent Medicine Trials Network for HIV/AIDS Interventions (ATN), creates an infrastructure to share and disseminate best practices in technology-based HIV interventions ( 41 ). Leveraging collective experiences in multiple app-based interventions facilitated development of the ePrEP platform after a 1-year pilot test ( 40 ). Researchers may also consider alternative funding venues, such as foundations, that may be more flexible and responsive to the needs of app development for health research.

A separate avenue is collaboration with private enterprises that have already achieved scale in the app space. Such an approach brings substantial benefits but also limitations. In favor of this model are the likely scalability and sustainability of the work. Additionally, by being on the cutting edge of what is in the marketplace, successful app businesses have access to the newest and most innovative tools that may benefit the research. This approach, however, requires meshing the business interests of the private enterprise with the research and public health interests of academic partners. Traditional research frameworks may need to be eschewed in favor of implementation science and monitoring/evaluation frameworks. Vested business interests may limit the scope of research, and issues of technology transfer and ownership may complicate relations. To the extent possible, these should be explored prior to commitment of the partnership.

Oversight of app technical development

Inclusion of an independent and research-versed developer to the research team can directly address the problem of information asymmetry. Many universities have technology-based app development groups that can provide such oversight through collaboration or trusted independent developers. Expertise is required early and periodically throughout the project. Early on in a project, an independent expert can ensure that the initial scope of work is sufficiently detailed to provide coverage of all program needs. The added costs of including an independent developer on the research team is often well worth the upfront cost because it can minimize potential problems regarding the scope of work and oversight of project development.

Independent expert oversight should extend to code architecture and development over the life of the project. Proper code architecture is essential for the performance of the basic tasks including (I) maintenance required due to phone operating system changes over time, (II) updates to address the appearance of unexpected issues as apps are used, and (III) addition of features demanded by users or researchers. Improper architecture can lead to results that confound and frustrate researchers unfamiliar with development: a simple bug fix can make seemingly unrelated parts of the app completely nonfunctional, requiring further fixes, a process that can spiral into cycles of dysfunctions and unexpected costs. These problems can also impact study outcomes if they occur during the course of a clinical trial. Moreover, an app designed with poor or highly stylized architecture may only be accessible for work by the original developers. This creates problems if the original developer either goes out of business or decides to substantially increase their prices. Proper planning and oversight can limit these pitfalls by ensuring that, as code is developed, it can be easily worked on and maintained by a coding team independent of the developer, with an architecture favorable to research data access and future app updates.

Structural changes

Structural changes could be made to capitalize on app-based research. NIH funds a number of center grants, and a future center could focus on expertise in health app development. Such a hub could anchor key contributions, such as (I) development of an open-source coding platform to address the most common research needs or (II) providing an at-cost service center model to provide expert oversight of code architecture for NIH-funded research projects. Adding a greater resource base could facilitate development of common functionality for open-source coding platforms that already exist, such as ResearchStack for Android, ResearchKit for iOS, or frameworks that allow simultaneous development of both Android and iOS such as React Native. Additional resources could include standardized consent or programming interfaces for commonly used external services such as survey platforms. Having these openly available to researchers could provide substantial resource savings for future app development. These efforts face challenges, however, such as staying current due to rapidly changing technology and norms of development. An alternative strategy could be to develop a model to facilitate app development for research across NIH. A hub where researchers share best practices could serve as a service model to provide key oversight or other functionality requested by NIH-funded researchers, and to document and disseminate best practices of and theory-based findings from app research.

To enhance the scale of evidence-based app interventions, it may be necessary to have privately or publicly funded dissemination programs. The US Center for Disease Control’s (CDC) Evidence-Based Interventions for HIV Prevention (EBI) is a useful example of such a program. The EBI program currently houses over 120 interventions that have been demonstrated to reduce HIV transmission or improve care outcomes for those living with HIV. Dedicated funding is used to disseminate these interventions, which for non-electronic programs has included development of intervention materials such as binders, pamphlets, videos, and other printed materials as well as ongoing provision of technical assistance. None of these EBI interventions are solely or even predominantly app-driven. A similar program for evidence-based apps could provide substantial utility, with translation and maintenance costs borne by the program. Such a program could be supported by CDC or alternate funding sources such as the National Institutes of Health or private foundations. Structured and dedicated funding would allow and empower health departments and community organizations to access evidence-based app programs, and would facilitate the visibility of app interventions via their inclusion in a publicly-funded, evidence-based compendium.

Conclusions

Mobile app health research is a promising avenue for health promotion, yet its implementation comes with many new challenges. We describe some of these and discuss strategies to address them. Investing resources in app research may facilitate its development, impact, and dissemination, allowing it to fulfill its substantial promise.

Acknowledgments

Funding: This work was supported by the National Institute of Mental Health (R01MH114692), the Adolescent Medicine Trials Network for HIV/AIDS Interventions (Protocol 159) from the National Institutes of Health (U19HD089881), and the National Institute of Allergy and Infectious Diseases (R01AI143875). The work was facilitated by the Emory Center for AIDS Research (P30AI050409). The contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/ .

Provenance and Peer Review: This article was commissioned by the Guest Editor (Lisa Hightow-Weidman) for the series “Technology-based Interventions in HIV Prevention and Care Continuum among American Youth” published in mHealth . The article has undergone external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/mhealth-19-263 ). The series “Technology-based Interventions in HIV Prevention and Care Continuum among American Youth” was commissioned by the editorial office without any funding or sponsorship. LHW served as the unpaid Guest Editor of the series. Dr. Hightow-Weidman reports grants from NICHD, during the conduct of the study. The authors have no other conflicts of interest to declare.

Application Development Research Based on Android Platform

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Advertisement

A Study of Mobile App Use for Teaching and Research in Higher Education

• Original research
• Open access
• Published: 05 June 2022
• Volume 28 , pages 1271–1299, ( 2023 )

Cite this article

You have full access to this open access article

• Annika Hinze   ORCID: orcid.org/0000-0002-7383-1134 1 ,
• Nicholas Vanderschantz 2 ,
• Claire Timpany 2 ,
• Sally Jo Cunningham 1 ,
• Sarah-Jane Saravani 3 &
• Clive Wilkinson 3  

19k Accesses

7 Citations

Explore all metrics

The exponential growth in the use of digital technologies and the availability of mobile software applications (apps) has been well documented over the past decade. Literature on the integration of mobile technology into higher education reveals an increasing focus on how mobile devices are used within the classroom environment, both physical and online, rather than on how mobile applications may be used for either teaching or the research process. Our study surveyed staff and higher degree research students at a New Zealand university using an online questionnaire to gain insight into the use of mobile apps for tertiary teaching and research, seeking information, particularly on which apps were used for which tasks and what obstacles hindered their use. The online survey used 29 questions and ran in 2016/2017. 269 participants completed the survey, nearly 20% of the potential sample. We found that mobile apps were used by academics and students for both teaching and research, primarily in the form of document and data storage and exchange, and communication. Very little app use was recorded for in-class activities (teaching) or in-field activities (research). Apps use resulted from personal motivation rather than institutional planning. Both students and academics reported that institutional support and flexibility would likely provide motivation and lead to increased app use for both research and teaching.

Similar content being viewed by others

Use of Mobile Apps for Teaching and Research – Implications for Digital Literacy

Mobile Technology for Learning During Covid-19: Opportunities, Lessons, and Challenges

Moving Towards the Effective Evaluation of Mobile Learning Initiatives in Higher Education Institutions

Avoid common mistakes on your manuscript.

1 Introduction

Mobile learning has been claimed as the future of learning (Bowen & Pistilli, 2012 ) yet surprisingly little specific empirical investigation of mobile application use in tertiary settings is available in the literature. While digital devices are prevalent in the higher education environment, the use and uptake of mobile apps for tertiary teaching and research by academic staff has only begun to be studied (Lai & Smith, 2018 ; Shraim & Crompton, 2015 ).

1.1 Technology Availability to Students

The 2019 ECAR survey of Undergraduate Students and Information Technology found that students see technology as a means for better engagement with study material, instructors and peers in the classroom (Galanek, & Gierdowski, 2019 ). The 2020 survey found that 75% of students who connect to campus WiFi are using two or more devices (Gierdowski et al., 2020 ). The 2018 survey reveals 95% of students have access to smartphones and 91% to laptops (Galanek et al., 2018 ). The downloading of mobile software applications (apps) in recent years shows a similar pattern of increase, rising from 84 billion downloads from the Apple App Store/Google Play in 2016 to 105 billion in 2018 (Sensor Tower). The third most popular Apple App Store category in May 2019, was education at 8.52% (Statista, 2019 ). Studies of higher education students in Southeast Asian universities reveal even higher percentages, for example, 100% of Hong Kong undergraduates in a 2018 study possessed mobile phones, of whom 85% also used apps for their academic studies (Shuk Han Wai et al., 2016 ). Thus previously held concerns that not all students will have access to a smartphone is not supported by the wealth of recent research investigating technology availability (Anderson, 2015 ).

For some time there has been the suggestion that technological advancement of mobile devices and the increased availability of mobile apps may prove central to academic teaching and research (Hahn, 2014 ; Canuel & Chrichton, 2015 ; MacNeill, 2015 ). Specific empirical investigation that discusses mobile app use as opposed to mobile device or more generally information technology use in tertiary teaching or research is extremely limited. Of the few specific discussions of mobile app use in academia, we identify library studies that have been conducted on the selection, use or development of mobile apps (Wong, 2012 ; Hennig, 2014 ; van Arnhem, 2015 ). These studies have often had a focus on the delivery of information or data about library services. Practitioner research in library and education have also included work describing apps and app features for research or teaching—an example being apps for ethnographic field research (van Arnhem, 2015 ). Work has investigated undergraduate student perceptions of mobile apps and mobile devices. An early study of tertiary student use of mobile note-taking software by undergraduate students (Schepman et al., 2012 ) saw widespread positive perception and adoption of these mobile tools by students. Studies exploring the impact the integration of mobile computing devices is having on higher education teaching and learning reveal an increasing engagement with content, collaboration with classmates and information creation and sharing outside the formal learning spaces (Bell et al., 2019 ; Compton & Burke, 2018 ; Gikas & Grant, 2013 ). Systematic literature reviews (Burch & Mohammed, 2019 ; Singh & Hardaker, 2014 ) and reports or investigations of academics’ perspectives of technology use in tertiary classrooms (Galanek & Gierdowski, 2019 ) provide insights into the broad picture but have provided little advice regarding app use for research or teaching.

1.2 Technology Use in Academia

Research on the integration of mobile technology in higher education is focussed on how mobile devices are used within the classroom environment, rather than on their application to the research process (Morris et al., 2016 ; Pedro et al., 2018 ; Schepman et al., 2012 ; Shuk Han Wai et al., 2016 ). MacNeill ( 2015 ) outlines techniques and strategies for the use of apps to support learning, teaching and research. The perspectives are self-reflective and provide insights into tools that have been trialled by the author with recommendations for educators to dedicate time to explore the wealth of available applications for teaching and research inside and outside the classroom. In the higher education classroom, mobile devices in higher education can provide new opportunities for information gathering and use, content access, communication, collaboration and reflection (Beddall-Hill et al., 2011 ; Bowen & Pistilli, 2012 ).

Lai and Smith ( 2018 ) identify a paucity of research on technology use in higher education. We identified two previous surveys of mobile technology use in tertiary teaching and learning. They focussed either on how socio-demographic factors influenced the perception of teaching staff (Lai & Smith, 2018 ), or the perceptions of the pedagogical affordances for mobile devices in teaching (Shraim & Compton, 2015 ). The survey of 308 tertiary teaching staff by Lai and Smith ( 2018 ) found that while many of the respondents were positive about the benefits that mobile technology could provide for their teaching, many felt they lacked the confidence to apply the technology effectively. “When implementing a mobile application in curriculum, instructors need to clearly state the goals of using the application to make sure the students understand the purpose of using the application for coursework, how it is connected to the curriculum, and how it will improve their learning” (Chen et al., 2013 , p.339). Other surveys of academics on their use of apps and mobile devices have focussed on teaching. The survey of faculty members use of mobile devices for teaching by Shraim and Crompton ( 2015 ) found that there are positive perceptions of the opportunities that mobile devices provide for teaching, but were focussed on the opportunities that the device itself provided (mobile connectivity, linking of formula and informal teaching, increasing enjoyment and connecting to real-world problems), rather than apps. The most important finding related to app use was the concerns that academics held about finding time to select appropriate apps and develop their teaching plans to incorporate them (Shraim & Crompton, 2015 ). Their scope was wider than app use, but only asked academics about their app use in their teaching, not their research.

Mobile devices provide opportunities to undertake research and fieldwork while enabling the collection, manipulation and sharing of data in real-time (Beddall-Hill et al., 2011 ). To date, the investigation of digital tools for research has focused on opportunities and challenges such as technical issues (e.g. battery life, data security or data inaccuracies) and considerations such as the preparation of future researchers to leverage the capacity of digital tools for research (Carter et al., 2015 ; Davidson et al., 2016 ; Garcia et al., 2016 ; Raento et al., 2009 ). The benefits of using mobile devices for research are described by Chen ( 2011 ), as including; immediacy of response, better enablement of longitudinal research, capturing of location information for context and the inclusion of an additional touchpoint to provide a more well-rounded research picture. Carlos ( 2012 ) suggests that mobile devices and mobile applications provide three main benefits for use in research: ready availability and familiarity, easy use, and always-on internet connections. A counter perspective is provided by McGeeney ( 2015 ) who observed a number of constraints for using mobile apps, compared to Web browsers. They found lower response rates, increased costs, and usability issues such as limited navigation and data entry options in mobile survey tools. Similarly, it is suggested that with mobile apps the time and effort required to learn how to use an app effectively can result in lower response rates than web-based data collection (Pew Research Center, 2015).

1.3 Institutional Expectations and Support

Many Institutions and academic libraries encourage mobile device use in educational contexts (Canuel et al., 2016 ; Hanbridge et al., 2018 ; Morris et al., 2016 ). Academics are encouraged to provide learning experiences that include “mobile-friendly content, multi-device syncing, and anywhere/anytime access” (EDUCAUSE, 2019 , p. 8). However, the 2019 Horizon Report has identified a need for sustained support and professional development to take advantage of the new teaching opportunities afforded by digital devices (EDUCAUSE, 2019 ). While academics are largely confident with mobile technologies, they need greater awareness of how these technologies can be incorporated effectively to take full advantage of the affordances mobile devices can offer in teaching ((Shraim & Compton, 2015 ). Several studies found that a lack of faculty training was a source for faculty dissatisfaction with classroom technology (Galanek & Gierdowski, 2019 ) and (mobile) IT integration into teaching (Burch & Mohammed, 2019 ; Shraim & Crompton, 2015 ). A number of academic libraries promote the use of mobile software to academics through digital or technological literacy training (Canuel & Chrichton, 2015 ; Hennig, 2014 ). However, research in the area of mobile application in academic libraries almost exclusively focused on the delivery of library services to mobile devices (Aher et al., 2017 ; Breeding, 2019 ; Singh Negi, 2014 ) or the integration of responsive design in web-based service (Kim, 2013 ; Tidal, 2017 ). A sample scan of university library websites indicates that it has become increasingly common for research university libraries to include guidance and instruction on the use of mobile apps for research. Such guidance usually takes the form of a brief preamble followed by a list of the various apps with a brief description of the features, functions and purpose of the app with links to the vendor website. Contextually, little indication is provided as to how or why such a list was curated or, more importantly, how the library supports the integration into learning of such mobile apps through training or instruction. A notable exception is the service offered by Stony Brook University Library, which assists in selecting and using mobile apps for research (Saragossi et al., 2018 , p. 202).

1.4 Research Questions and Focus

We identified a number of shortcomings in the existing literature on app use in tertiary contexts. Research on technology use in teaching and learning rarely focuses on (the experience of) app use, but rather on device capabilities and opportunities of technology use. Previous surveys predominantly analysed the undergraduate students’ perceptions of app use in teaching. Use of apps for academic research is little discussed beyond app use for specific projects, and general technology benefits or issues. While many tertiary institutions actively encourage academics to use mobile apps (and other technology) for teaching, the impact of such expectations on the academic experience is not well studied.

The research reported here attempts to understand more widely how apps are being used in tertiary teaching and research, including what are the perceived benefits and barriers. To provide insights into how mobile apps may be used by students and staff in teaching and research a university-wide survey on mobile app use in a tertiary setting was conducted. The survey design was guided by the following research questions:

RQ1: Are academics using mobile apps for tertiary teaching and research at University of Waikato? RQ2: Which apps are used by academics for which teaching and research tasks? RQ3: What is the experience of app use by academics: what obstacles/opportunities do they identify?

The survey was made available to staff and higher-degree students (collectively referred to as academics) across the University to capture their perspective on mobile app use for teaching and research.

This article presents our study data, and analyses these with respect to the three research questions posed above. The remainder of this article is structured as follows, in Sect.  2 we introduce our method, an online survey of staff and higher degree students at a New Zealand university. Section  3 provides results and analysis of the responses to this survey. We discuss the findings in light of our research questions in Sect.  4 and conclude this article in Sect.  5 . Initial analysis results were presented elsewhere (Hinze et al. 2017a , b ), and primarily focussed on the responses from higher degree research students. The results reported in this paper cover all responses to the survey including those of higher degree students and staff.

We performed an online survey of staff and higher degree students of the University of Waikato in New Zealand. The survey was designed to get a university-wide view of how mobile apps were being used for teaching, research, and learning purposes. The survey was performed over two consecutive years in order to capture the widest sample of participants.

2.1 Context of Study Environment

This New Zealand University is typical of western universities offering qualifications across multiple academic divisions including, but not limited to; the arts, computing, education, management, and the sciences. The majority of staff and students work on campus yet mobile and electronic learning is supported at all learning levels. The university provides Google apps for email, file storage, and word processing. A number of digital resources and technologies are supported depending on the needs of researchers and teachers in academic disciplines. A well-resourced library supports students and staff with print and electronic holdings. There are no required or mandated mobile apps at this university.

2.2 Data Collection

A location-restricted online, self-administered survey tool was developed in the Qualtrics Survey Software. The survey was made available to participants at the University of Waikato in New Zealand from 3rd to 19th August 2016 and again from 31st August to 6th October 2017. The potential sample size was approximately 820 enrolled masters or PhD thesis students and 580 staff (including academics, researchers, and research administrators). All responses were anonymous.

2.3 Participant Recruitment

Higher-degrees students and staff from across the university were invited to participate. We engaged the University’s research office to forward invitations to all departmental administrators, with whom we personally followed up with to distribute the survey invitation to all the University’s academic staff and researchers via email. We further followed up these email invitations with in-person invitations by one of the research team at Faculty and School meetings. The higher-degree students were engaged by the School of Graduate Research through email and social media. Our study had a potential pool of 1400 staff and higher-degrees students.

The survey was done in two stages (same target group, self-selected participants, initial and repeat attempt to engage participants), we present in this article the aggregated result of both stages. 288 survey entries were received, out of which 19 contained no further data and were excluded from the analysis. The survey was thus completed by 269 participants, or nearly 20% of the potential sample of university staff and higher-degree students.

2.4 Survey Tool

Our online tool was a 24-item survey that incorporated a combination of Likert scale tools, radio button responses, and free text questions. This tool was conceptualised in three sections which (1) requested demographic data, (2) surveyed previous experience and use of mobile apps, and (3) reviewed device and operating system use. The survey invited reflection by the participants on their use of mobile apps and whether they believed that their use or lack of use had influenced research or teaching practice. The survey also required participants to give information regarding their reasons for non-use in cases where participants indicated that they had not used, and were not intending to use, mobile apps. To review the survey questions please refer to Appendix 1.

2.5 Definitions Used in the Survey

In the survey we included the following definitions for clarity for the participants:

Mobile app—is a software application developed primarily, although not exclusively, for use on small computing devices, such as smartphones or tablets. Examples include WhatsApp, Evernote, and Flipboard. Other examples might include mobile app versions of programs such as Dropbox or EndNote.

Academic purposes—includes all teaching and/or research activities engaged in while a member of the University community.

2.6 Data Analysis

The results were analysed using default and cross-tabulation report functions provided by the Qualtrics software before manual manipulation, tabulation, and analysis using Excel. We have undertaken basic descriptive statistical analysis (means testing and T test for cohort comparison) and provide tables, graphs, mean values and probability values (where appropriate) along with our reporting in the Results section.

3 Results and Analysis

We present our results structured by the three research questions. After demographic information in Sects. 3.1 , 3.2 , 3.3 address the first question ( are academics using mobile apps for tertiary teaching and research) , while Sects. 3.4 , 3.5 address the second question ( which apps are used for tasks ), and finally Sects. 3.6 , 3.7 , 3.8 address the third question ( academic experience of app use: obstacles and opportunities ).

3.1 Demographic Attributes

The university staff and postgraduate students at the time of the two instances of the survey was reasonably stable at about 1400 (580 academic staff and 820 higher-degree research students), which forms the potential participant pool. 269 of these 1400 responded to our invitation, with most of our study participants being academic staff (N = 163), followed by doctoral students (N = 83), see Fig.  1 .

Participant roles (multiple selections possible)

Out of the 269 participants, 141 were female (52%) and 125 were male (46%); 2 did not specify gender (1%), and 1 selected other.

63% of the participants were younger than 40 years old, see Fig.  2 . The participants represent a range of schools and faculties, as shown in Fig.  3 . The other university areas mentioned by participants were administration and technical support. Five participants selected two options.

Participant age

Participants by school/faculty (multiple selections possible)

3.2 Use of Mobile Apps

With 172, the majority of the 269 participants (64%) had used mobile apps for academic purposes such as teaching or research, see Fig.  4 for details. We note that the percentages among Academic staff and doctoral students were comparable at 67% and 69%, respectively, while only 25% of Master’s students had used apps for research. Four participants provided no data (Fig. 5 ).

Prior use of apps for academic purposes (multiple roles possible)

Academic use of mobile apps by participant age range

Of the 172 participants who had used mobile apps for academic purposes, the age cohort that showed the strongest engagement were the 21–30 year-olds (71%). This was followed by the group of 31–40 year-olds (64%). If broken down by gender, 62% of the 141 female participants and 67% of the 125 male participants had used apps for academic purposes ( p  = 0.3975, i.e., there was no significant gender difference in app use), see Fig.  6 .

Academic mobile app usage by participant gender

Out of the participants who had used apps for academic purposes, most (19%) were in the Faculty of Computing and Mathematical Sciences, followed closely by both the Faculty of Education (18%) and science and engineering (18%); details are shown in Fig.  7 .

Academic mobile app usage by participant school/faculty

We surveyed the 172 participants who had used mobile apps for academic purposes to inquire which types of devices they used mobile apps with (multiple selections were possible). 303 responses were collected. The majority (79%) used smartphones, followed by iPad and Android tablet devices (together 70%), details see Fig.  8 . The named other devices were laptops and PCs, and one sporting device.

Type of mobile device used (multiple selections possible)

90 of the 172 app users gave details about operating systems with 114 selections; for details see Fig.  9 . Under ‘Other’ participants listed ChromeOS and Microsoft system (surface tablet). As expected based on mobile phone ownership data, Android and iOS emerged as the preferred operating systems.

Operating system used on mobile device (multiple selections possible)

Finally, we also asked if participants had been involved in the development of any mobile apps that might be used for academic purposes, and to explain their purpose. We received 60 answers: 50 no, 5 n/a, and the 5 positive answers: driving support (1), for teaching (2), indigenous language learning (1), and a personal digital library (1).

3.3 Purpose of Mobile App Usage

In order to investigate the mobile app use-cases in the tertiary environment, we asked participants about the situations that they had used these. Participants could select either or both teaching/supervision, and/or research. Ninety-five (56%) of the 171 respondents to this question had used a mobile app for teaching/supervision purposes; 146 (85%) had used one for research purposes. Of these, 70 (41%) selected that they had used apps for both (see Fig.  10 a, top).

Mobile app usage: ( a) by purpose (top), ( b) by gender (bottom)

More female participants are using apps than male participants (see Fig.  10 b, bottom). For teaching, there was not significantly more male respondents using apps than female respondents ( p  = 0.96). For research, more female participants were found to be usings apps than male participants, though this was still not significant ( p  = 0.69). We further note that female respondents tended to use apps for research or for teaching only (63.2% of 87 female compared to 54.7% of 84 male). Conversely more male respondents used apps across both categories (marked in gray). However, the difference between male and female use of apps for both purposes was not significant ( p  = 0.59). The majority of the participants who had used apps for teaching or supervision were academic staff (86 of 95). A small number of participants who had used mobile apps for teaching identified as doctoral students (15 of 95), none as Master’s students, 7 as Other (multiple selections possible). 88 Academics, 55 Doctoral students, 2 Master’s students and 18 Others reported using mobile apps for research purposes.

We observe that higher percentages of academic staff used a mobile app for teaching and supervision purposes compared to research purposes (see Fig.  11 ). Conversely, doctoral students were more likely to use apps for research purposes than for teaching/supervision purposes. Quite predictably, Master’s students and other participants were more likely to use mobile apps for research.

User roles for mobile app users

Only 6 participants reported being asked by their lecturer or supervisor to use mobile apps for academic purposes (50 reported having not been asked, 214 provided no answer). They named the following app purposes: document sharing, storage, referencing, communication; bookshelf app for recommended lecture text; conference presentation app; google drive and dropbox for backups of theses, and app examples to explore for research on interactive tour guides.

3.4 Apps for Teaching/Supervision

Ninety-five participants reported using mobile apps for academic purposes for teaching or supervision related activities. Unsurprisingly, the majority of these participants reported themselves as teaching staff. At this university, it is not atypical for staff to work across roles in a university, and for some higher degrees students to be contributing to teaching initiatives at various levels and therefore some doctoral students and participants in the ‘Other’ category had also used apps for teaching purposes. These 95 participants were asked to select from a shortlist of possible academic-related apps (see Fig.  12 ) the mobile apps that they used for teaching or supervision purposes. Also shown in Fig.  15 , the participants were asked if these apps were used by themselves or by students under their supervision. There was a substantial number of Other options named, including Google Drive (8), Google Docs (5), Facebook (4), Google Sheet (3), Kahoot (3), and Kindle (3) and a further 11 programmes named twice, and 65 programmes named once showing that a diverse range of apps were used (not shown in Fig.  12 ).

Apps used for teaching/supervision (multiple selections possible)

Mobile apps for teaching purposes were reported as being used by 95 participants, the specific purposes for using apps for teaching are elaborated on in Fig.  13 . The aspects teaching staff most engaged in were sharing or storing documents, as well as communication with colleagues. Other tasks mentioned were communication with students, in-class surveys, or keeping up with recent blogs.

Use of mobile apps in teaching practice (multiple selections possible)

There were 95 participants that had used a mobile app for teaching/supervision, of which 71 had requested their students to do the same. These participants were asked to state the purpose for making this request; results are summarised in Fig.  14 . The responses in the ‘Other’ category included quizzes, vocabulary practise, feedback, class activities, creative practice. Figure  14 shows that the primary reason for asking students to use mobile apps was for the purposes of communicating with others, sharing documents, followed by accessing course information.

Mobile apps recommended to students (multiple selections possible)

3.5 Apps for Research

Of the 172 participants who had used mobile apps for academic purposes, 146 did so for research purposes (85%), one participant provided no answer. This group of 146 participants were asked what academic-related mobile apps they had used for research purposes from a list of possibilities provided. The results, summarised in Fig.  15 , show the file-hosting app Dropbox was extremely popular and used by 62% of researchers (N = 91). There was a substantial number of participants (65) who provided ‘other’ options, with many participants naming up to 6 or 7 apps, including Google apps (N = 22, among which were Drive: 12, Docs: 3, Keep: 3, Slides: 2, Gmail: 3), Mendeley (N = 5), Skype (4), voice recording (3), Twitter (3). Participants also mention apps that had been written by themselves or their students.

Mobile apps used for research (multiple selections possible)

Participants were also asked what research purposes they used mobile apps for (see Fig.  16 ). Storage and sharing of documents, as well as searching and note-taking were the main reasons for researchers using mobile apps. Only 22 ‘Other’ answers were collected, mostly naming different uses such as reading (6), recording of various data, such as interviews (2) and notes on whiteboards (1), and app development (2).

Purpose of mobile app use for research (multiple selections possible)

3.6 Impact of Apps on Academic Experience

All participants who had indicated that they used mobile apps for academic purposes were asked to respond to questions on their use of mobile apps for their teaching/supervision or research, their knowledge of apps, and their use of mobile apps. The response required from participants was on a 5-point Likert scale from strongly agree (1) to strongly disagree (5), see Fig.  17 . The factors that participants reported to most strongly agree with was “my research or teaching benefited from the use of mobile apps” (mean = 1.72) and they “had no problems finding a suitable app for my research or teaching” (mean = 2.40). The attitude statement that participants most strongly disagreed with was “I experienced difficulties in using mobile apps” (mean = 3.60). Other responses regarding the attitude towards app use were; “the outcome of my research or teaching was impacted by the use of mobile apps” (mean = 2.49), “my research or teaching practice was conducted differently as a result of using mobile apps” (mean = 2.53), and “I know where to go to get help with mobile apps” (mean = 2.62).

Attitude to mobile app use: data out of 100% = 172 participants

Only 20% of participants experienced difficulties when using mobile apps in an academic setting. 45 to 60% of participants knew where to seek help and where to find suitable apps (vs 15–25% who did not; 7% no answer). A similar observation holds for the perceived impact of using apps for research and teaching both in terms of change of practice and outcomes.

However, nearly 90% of mobile app users responded that they felt they had benefited from, or felt neutral about, the inclusion of mobile apps in their academic activity (2% slightly disagreed, 8% no answer).

3.7 Experience of Users

The survey provided an opportunity for participants to provide any further comments they wished on mobile app usage in an academic setting. 60 participants provided comments, 18 from participants who had not used apps for academic purposes, and 42 from participants who had experience with such apps. Participants will be referred to by identifiers P1 to P269. Many of the concerns voiced were brought up by non-users and users alike. We, therefore, do not discuss their comments separately but indicate which category a participant falls into next to the identifier (P U for users and P N for non-users).

In comments provided by non-users, distrust in app/technical reliability were expressed, such as by participant P N 124: “Technology moves so fast that planned obsolescence is commonplace. New apps have a track record of failure in their first years: this does not look good to students if suddenly the app for their course falls over”. Similarly, P N 146 comments “I wish people would switch their bloody mobile phones off, and get a life really.”, and P N 231 “I do not have a mobile”.

Participants also discussed mobile app usefulness from a pedagogical viewpoint , stating that “[…] we have gone into more and more web-based teaching, and moodle etc. However, I have seen that … students who will end up as designers in some companies do not gain much from these approaches. In my judgement and experience … use of white board and limited amount of notes uploaded will work well, with [a] lot of laboratory type hands-on elements. I strongly believe that if we [lose] the 'human touch" in [the] classroom setting, it will gradually and negatively affect the quality of the graduates we produce” (P N 128).

Several users commented that they are planning to do more or feel still at the beginning of their journey and wish for more support : P U 233: “I've been reluctant because of time, planning and other flexibility related restrictions it places”, P U 254: “Most of the learning on this is on my own. more exposure is needed through seminar etc.” P U 87: “Would be great to get some training on this)”. Some expressed reservations about institutional support, for example, P U 108: “Help with mobile apps seems to be largely found in internet searches of forum posts and vendor provided documentation”.

Participants expressed that guidance on choosing apps was needed as “It would also be great if there was some sort of online resource on the uni website that lists and briefly explains some of the apps that might be useful when conducting research” (P U 39), and the concern that “There is simply not the capacity in ITS to support mobile app usage” (P N 124). Similarly, non-users wished for more support: P N 108: “Help with mobile apps seems to be largely found in internet searches of forum posts and vendor provided documentation”,

Some participants considered app use inconvenient , claiming “In many instances and situations a well thought out website enhanced for use on mobile will be more useful and less cumbersome than an app. I despise having to download and constantly update several apps, plus they come with intrusive permissions” (P71). Or participants felt that apps were “only useful where use of a real computer is impossible”. The context within which apps could be integrated into the learning environment caused some uncertainty, with several comments highlighting this reservation, “It is sometimes challenging to find the most appropriate app to meet a specific teaching purpose” and “The challenge will be to develop apps or modify existing apps to suit the purpose of the user and the context of the user”.

Finally, some participants expressed a dislike or unfamiliarity with/for phones and technology in general: “I wish people would switch their bloody mobile phones off, and get a life really [..]” (P N 146) and a distrust in apps as they expressed concern that “they need to be reliable enough that researchers can be confident that they will not suffer data losses if they use just apps” (P U 105). Similarly, worries about the hardware were expressed: “Our devices need updating. Phones are personally owned and my ipad is too old for some of the apps I want to use.” (P U 155).

Some comments seemed to be expressions of undisclosed fears that were channelled into the reasons given. For example, P U 104 raised the issue that “One can only move as fast as students are able. One can only do so much introducing of new technology—you can get to a point where you have built a learning task for example on a particular resource and then find that half the class cannot even access it”.

A theme that was detected in the responses received reinforced the mobile nature of both tertiary education and academic publishing today. This can be specifically seen in the discussion of mobile and on-the-go teaching, learning, and research. Participants listed the importance of being able to collect data, take notes, as well as communicate with peers, participants, and users in a variety of situations. One participant noted, “I've largely found it useful for mobility rather than anything else.” Another participant, whose complaint we noted earlier regarding screen size making viewing information less pleasurable for them compared to a computer, did note “at least information is available and accessible when on the move”. A further PhD student stated that “mobile apps are great. If you are in tedious work meetings you can work on easy bits of your thesis and people just think you are diligently taking notes”.

Similar numbers of participants believed their work-life was or was not impacted by mobile apps as participants who believed their teaching or research practices were different today because of their mobile app use. Investigation of the impact of technologies including mobile devices and applications on traditional pedagogies and research practices and processes warrants further empirical investigation.

Significant discussion related to use of apps for teaching rather than research. With some being enthusiastic: “We are moving into the new generation Apps is the tool to connect with the students. / Let’s not hesitate. We need to be engaging successfully to create a sense of new age.” (P U 81), while others are quite reserved about technology use, including “ web based teaching, and moodle” (P N 128). P U 78 described challenges: “It is sometimes challenging to find the most appropriate app to meet a specific teaching purpose”. P N 203 teaches online papers and comments “it would be great to have a way for students to access discussion groups and to have virtual communication through a mobile app”.

Several participants explicitly wished for apps that allowed access to library resources such as eBook readers (P U 33, P U 7), library search (P U 42, P U 120), and a personal library (P U 202). Some participants were very enthusiastic about the potential of apps in the academic environment, such as “We are moving into the new generation of Apps is the tool to connect with the students. Let’s not hesitate. We need to be engaging successfully to create a sense of new age” (P U 81) and “Apps greatly increases my ability to store quotes and research links” (P U 67). Conversely, some participants used the open feedback option to comment on the shortcomings of their personal phones (P N 169: “I find the real estate of my mobile device is too small [..] my tablet is too slow”), on perceived shortcomings of innovation management (P N 182: “endless workshops”) or even expressed fears about the motivation of the survey (P N 166: “The outcome of studies like this can be deeply political”); conveying a sense of fear about potentially being forced to use apps for research and teaching.

3.8 Experience of Non-Users

In Sect.  4.2 we report that 35% of the participants had not used mobile apps for academic purposes (N = 93). More than half of these non-users (N = 50/93) indicated that they did not intend using mobile apps for academic purposes in the future. Forty-four percent (41/93) of these non-users reported that they do plan to use apps in the future. We asked non-responders what their reasons for non-use of mobile apps in academic contexts were. Forty-seven people responded to this question. Nearly half of the 47 participants reported they lacked knowledge about how they might use mobile apps for their purposes. Further to this approximately one-third of the participants confessed their disinterest in apps, while approximately a third considered them to be irrelevant for their teaching or research needs. Eight participants reported a lack of apps for their purposes and 7 participants discussed the perceived lack of support from the university. Other opinions suggested that computers and large screen devices serve their needs better than mobile devices for academic purposes. One academic responder twice noted planned obsolescence as a factor hindering their use of mobile apps in the academic context. Some participants also named specific fields for which they believed mobile apps or small screen devices would not be suitable.

We noted earlier that 44% of non-users had indicated they might use mobile apps in the future. We asked these non-users to select what factors might influence their future use (multiple selections were possible). 40 participants responded to this question (1 provided no data) resulting in 145 selections (see Fig.  18 ). Non-users were most interested in mobile apps that supported them to share or communicate with others (selected 23 times), see Fig.  19 . The option ‘Other’ included participant sign-up, reading, engaging with students in and out of lectures (3), and the possibility of so far unforeseen usages (3).

Reasons for intended non-use of mobile apps (multiple selections possible)

Non-users intended future use of mobile apps

The 41 participants who reported not using mobile apps were asked how helpful the six factors shown in Fig.  20 might be in facilitating the uptake of mobile app usage for academic purposes. This question was posed as a 5-point Likert scale from very helpful (1) to very unhelpful (5), for which 38 of 41 people responded. Responses show that “more appropriate apps” (mean = 1.54) and “easier to use apps” (mean = 1.55) were the factors most likely to facilitate uptake with app non-users. This was followed by; “more practical support” (mean = 1.58), “more institutional support (mean = 1.66), “more information about apps” (mean = 1.66) and better access to appropriate devices (mean = 1.81).

Factors facilitating uptake of mobile apps

Of the six factors posed, the two that were defined as very helpful and helpful were factors relating to “easier to use apps” and “more appropriate apps”.

4 Discussion

We here discuss our findings in light of our research questions, their implications and opportunities. Our research was motivated by three questions, which we will answer here based on our study results. We will compare and contrast our findings with the related work, giving specific relation to two related surveys of academic use of mobile technology use (Lai & Smith, 2018 ; Shraim & Crompton, 2015 ).

4.1 Answering RQ1: Are Academics Using Mobile Apps?

Our study had 269 participants from a potential pool of 1400 staff and HRD students (19.2% response rate, including 28% academics and 10%). The two related surveys had similar response rates of 24% among teaching staff (Lai & Smith, 2018 ) and 29% (Shraim & Crompton, 2015 ), with similar distributions across gender (i.e. a slight to significant majority of male respondents).

172 of our 269 participants (64%) had used mobile apps for academic purposes such as teaching or research. The percentages among academic staff and doctoral students were comparable at 67% and 69%, respectively, but much lower for Master’s students. 95 of 172 (56%) had used a mobile app for teaching/supervision purposes; 146 (85%) had used one for research purposes; and 70 (41%) had used apps for both. By contrast, Lai and Smith ( 2018 ) found that the majority (75–90% for comparable categories) of their respondents had not used any mobile technology for teaching. Shraim and Crompton ( 2015 ) did not report previous app use for academic purposes.

We found that 62% of the 141 female participants and 67% of the 125 male participants had used apps for academic purposes. By contrast, Lai and Smith ( 2018 ) found that more female teachers used mobile technologies for teaching than male teachers. They hypothesised that the reason may have been that the female teachers were younger than the male teachers in their response cohort. They also found that junior teachers are more willing to learn to use new technologies than senior teachers. We similarly found the strongest engagement with mobile technology among the 21–30 year-olds (71%), followed by the 31–40 year-olds (64%). Shraim and Crompton ( 2015 ) noted that three-quarters of their respondents were aged between 25 and 45, going so far as to suggest that older faculty chose not to respond, perhaps being less inclined to use mobile technology as part of their teaching. Some of our participants were of a generation where the technology may be seen as a hindrance or unfamiliar tool. For example, participant P N 191 stated “I think strategic training is really necessary for people like myself who is not a digital native—what are the benefits? How to develop greater usage in daily work and life?” However, very few participants who had used apps did report technical difficulties (see our discussion in Sect.  4.3 ). We conclude that the study participants who did use apps for teaching and research were proficient, while the extent to which non-users experienced difficulties is hard to gauge.

Our findings support the related literature that academics are using mobile technology and mobile apps for teaching and research. These findings imply there is a need to more deeply understand the reasons for app use/non-use by academics across tertiary institutions. From there, an exploration can be started of how appropriate support can be provided.

4.2 Answering RQ2: Which Apps are Used by Academics for Teaching and Research?

Apps for Supervision/Teaching Participants reported app use for tasks that involve sharing or storing documents, as well as for communication with colleagues and students, and some use for in-class surveys, or keeping up with recent blogs. Similarly, it was reported that teachers required students to use apps primarily for communication and information storage or delivery purposes. This is in line with many studies that suggest that mobile devices in higher education may provide new opportunities for information gathering and use, content access, communication, collaboration and reflection (Beddall-Hill et al., 2011 ; Bowen & Pistilli, 2012 ). The tool that academics most reported as being used by themselves and by students was Dropbox, a file sharing and storing app that facilitates collaboration and information dissemination. Neither of the two related surveys (Lai & Smith, 2018 ; Shraim & Crompton, 2015 ) focussed on the use of mobile apps or specific software, but rather on technology use.

However, many participants reported a lack of time, resources, and control as reasons why they have not successfully implemented mobile apps into their teaching for use by or with students. Participant P U 233 noted “I've been looking at Kahoot at the like for teaching. I've been reluctant because of time, planning and other flexibility related restrictions it places”, while P U 154 reported “at the moment, I am just using the iPad to save paper. It hasn't really impacted how I teach. I am aware that there is far more I could do with it, but I do not have a lot of control over what/how I teach.” Lack of time, resources and knowledge are well-known issues for academic use of technology that were observed in other studies as well (Ajjan & Hartshorne, 2008 ; Lai & Smith, 2018 ; Shraim & Crompton, 2015 )).

Another interesting aspect was the perception that teachers need to restrict students’ screen time (P N 187): “with my overseas students (English language learners) … I try to promote personal f2f interaction in my lessons and try to get the young students away from their screens!” While this was not a prevalent theme, it deserves consideration in future research.

Apps for Research Sixty-four percent (172 of 269) of participants had used apps for research or teaching. A number of apps were listed for participants to select from. The participant was able to select multiple apps that they had used for their research. The research team had hypothesised a number of bibliographic, file sharing, and document creation tools for participants to select from. While file hosting and sharing was reported as being used significantly by participants, it was interesting to note that social media (Twitter), communication (Skype), as well as file creation and storage solutions (Drive, Google apps, voice recording) were also listed by numerous participants. If we consider the nature of research and international connectedness that is expected in universities today, it is unsurprising that a number of these apps that allow for asynchronous collaboration and long-distance telecommunication are listed as central to the modern research framework. This is summed up by one participant (P U 206) who commented “the survey seems to focus on information management. Apps also allow easy access to communication and collaboration channels.”

Higher numbers of academics and students reported using apps in the early phases of the research process for tasks such as note-taking (64 participants), search (66), research planning (59), communication (43), data collection (60), and document sharing (73), compared to later phases of the research process such as data analysis (21), presentation (30), and publishing (16).

App use for research was not considered in the related surveys (Lai & Smith, 2018 ; Shraim & Crompton, 2015 ). To the best of our knowledge, no comparable data has been collected so far. The implications of these findings is that work to support and develop appropriate mobile applications that service academics during all phases of the teaching and research process are required.

4.3 Answering RQ3: What is the Experience of Academic App Use?

We here discuss first the experience of respondents who had used apps, and then those of respondents who did not use apps but had identified obstacles.

Impact on Academic Experience The majority of our participants did not encounter any issues with finding and identifying relevant apps. Our participants also did not encounter major technical difficulties when using apps. For example, only three explicit comments called for technical support and only 20% of participants mentioned technical difficulties. Most observed that using apps influenced the way they did their teaching and research. The vast majority of mobile app users (90%) in our study felt that they had benefited from, or were neutral about, the inclusion of mobile apps in their academic activity. Both Lai and Smith ( 2018 ) and Shraim and Compton ( 2015 ) also explored teachers’ attitudes towards mobile technology use in the classroom but did not ask if teachers experienced the technology as having been helpful. Like most other publications on technology use for teaching (Hahn, 2014 ; Canuel & Chrichton, 2015 ; MacNeill, 2015 ), they asked instead about the teachers’ beliefs in the opportunity of enhanced learning, which may not align with the actual experience of using mobile apps. As a potential drawback, they named students becoming less critical, or increasing their workload (Lai & Smith, 2018 ). Given the low percentage of mobile technology use (< 25%), this feedback is largely not based on the academics’ experience. While they reported that their departments supported the use of mobile technology, it remains unclear if this describes a positive attitude or practical help (Lai & Smith, 2018 ).

Lack of support A common theme was a lack of support by the institution for mobile device and mobile app use for teaching, learning, and research purposes. Participants noted a need to be supported in identifying apps of relevance and suitability to their teaching and research. One academic participant discussed a “notable lack of support for adequate apps, a case in point being that the Uni does not provide apps suitable for reading online books” (P N 124). The results of both studies suggest that non-users may be more willing to use mobile apps if institutional support and guidance were provided. This desire for institutional support came from both academics and students, with higher degree student P U 33 reporting “it would be very beneficial to have an online list, or equivalent, of useful apps for students, varying from note-taking, referencing, data collection right through to ones specific to different fields of study. Many of the apps I now use would have been extremely useful had I known about them when I began this degree.” This reporting by academics of a need for institutional and wider support in selecting and using apps to support their pedagogy, classroom practice, and research is in line with Horizon Report Preview (EDUCAUSE, 2019 ) that calls for sustained support and professional development in order to take advantage of the new teaching practice opportunities afforded by the inclusion of digital devices within the education environment. A similar sentiment has been mirrored in other studies (Ajjan & Hartshorne, 2008 ; Chen et al., 2015 ; Lai & Smith, 2018 ; Shraim & Crompton, 2015 ).

Non-use 35% of our participants reported not having used mobile apps for academic purposes. Furthermore, approximately half these reported they had no intention to use apps in the future. Disinterest in mobile apps for teaching, or a view of mobile apps as being irrelevant to the participant, were common reasons for these responses. Some also noted a preference for desktop solutions for these tasks. This is summed up by P N 174 who noted “I don't like/prefer to use apps for academic purposes. I feel more comfortable on desktop/laptop when having to access content relating to academic needs”, while another participant stated “computers have more options than mobiles” and the perennial concern “screen size makes viewing information not as pleasurable as computer”.

Of the non-users, slightly under half suggested they might use or were willing to use mobile apps in the future for academic purposes. Non-users of apps were primarily interested in the potential ability to communicate or share with others. It is interesting to note that the communication affordances are of high interest because in both surveys the view that there is no use for apps besides for communication was a common criticism for mobile apps. Perceived potential benefits of mobile apps by non-users were features such as participant sign-up, reading and engaging with students in and out of lectures. Another feature that participants noted as a potential positive for mobile apps was the perceived convenience of managing, capturing, collecting, and storing information.

Many participants saw a need for future development, advancement, and indeed further research such as that we offer here. Almost all non-users identified “easier to use [apps]” and “more appropriate [apps]” as important or helpful. One participant summed this up “the challenge will be to develop apps or modify existing apps to suit the purpose of the user and the context of the user”, while another stated “I think [mobile apps] have some good potential for engaging students in classrooms and out of classrooms. I also don't think they are the be all, end all of engagement (i.e., necessary but not sufficient for good engagement).” There appears, in addition, to have been a perception that because software or apps are open source that they do not require coordinated technical support or training from the University. Through conducting this survey we found that there is a need to provide support and information to users for both subscription software as well as open-source alternatives.

The survey has highlighted that current users have typical usage patterns and generally feel confident with the use of mobile apps for a range of purposes. There was also a group of non-users and low-users that did not feel confident. We feel the implications of our findings are the need to support academics to locate and use mobile apps during teaching and research and the desire from academics for this support as well as for new mobile apps to meet their needs.

4.4 Limitations

This study was based at a single university in New Zealand; however, its results and recommendations for engagement and need for ongoing support are potentially widely applicable for a western tertiary education environment due to similarities in academic environments. One may expect differences in the specifics of apps used, such as the prevalence of Google tools in this sample, vs the use of OneDrive for similar tasks in universities with Microsoft contacts.

A participation rate is in keeping with typical response rates for similar online studies (Fosnacht et al., 2017 ; Nulty, 2008 ; Van Mol, 2017 ). As the participants were self-selected, it is unclear to what extent our sample accurately reflects the university situation. As our participants were self-selected, they did not necessarily constitute a representative sample of the whole university but rather reflected the feedback of people who felt strongly enough to engage in the process. While both users and non-users of mobile apps were explicitly targeted, the resulting sample consisted of predominantly mobile app users (66%). We hypothesise that non-users may have been less inclined to respond to a survey about app usage.

The study ran at the same time in two consecutive years. One notable difference was the proportion of academic vs student participants within the studies. However, there did not appear to be overall a significant variation between the results obtained in the first year vs the results from the second year, which were therefore presented here together. We noted that some participants commented also on the use of web applications. In order to keep the study results comparable, we did not change any questions. However, in future studies we would wish to include both mobile apps and web apps (i.e., software as a service), thus addressing the use of any software services away from the office or lab environment.

5 Conclusion

While students and academics use digital devices in the higher education environment, the uptake of mobile apps for tertiary teaching and research has only begun to be studied. Research on technology use in teaching and learning rarely focuses on the experience of app use by academics. The impact and experience of the institution’s expectations regarding apps by academics is not well studied. Our research attempts to understand how apps are being used in tertiary teaching and research, including what are the perceived benefits and barriers. Our study used an online survey, aiming to answer three research questions. Our study here is unique in that it has investigated students and academics’ attitudes to mobile apps in both the tertiary classroom and the research environment.

The contributions of our research presented here are the following: We conducted the first study into the experience of mobile app use for teaching and research by academics. Findings from our research are as follows: (1) Mobile apps were used by academics and students for both teaching and research, primarily in the form of document & data storage and exchange, and communication. Furthermore, the stated primary motivators for future mobile app use for both teaching and research were again the ability to communicate, collaborate and share with others. (2) Very little app use was recorded for in-class activities (teaching) or in-field activities (research). (3) Our study results and related work show that at present academics and students use mobile apps due to intrinsic personal motivations rather than institutional support or provision. There remain, consequently, opportunities for better support of mobile app use. (4) Both students and academics reported that institutional support and flexibility would likely provide motivation and lead to increased app use for both research and teaching.

Many of the apps named in our study were mobile versions of web apps (such as Dropbox, Evernote, Google Drive). Some participants may even have interpreted mobile app use to mean both mobile apps and web apps (e.g., for bibliographic software Zotero and Endnote). This interplay of mobile apps and web apps (or mobile access to web apps) has not been explored for the academic context so far and should be studied in a follow-up survey. Extending this survey with consideration of software as a service (SaaS) used on mobile devices may shine a light on some of these wider-reaching applications which also facilitate teaching and research.

Our study is the first of its kind, exploring the practical experience of academics using mobile apps for teaching and research. Data such as ours can inform academic management to better support students and staff with mobile app selection and use in the academic context.

Availability of Data and Material

Data not available for privacy reasons.

Code Availability

Not applicable.

Ajjan, H., & Hartshorne, R. (2008). Investigating faculty decisions to adopt Web 2.0 technologies: Theory and emperical tests. Internet and Higher Education, 11 , 71–80.

Article   Google Scholar  

Anderson, M. (2015). Technology device ownership: 2015. Report by Pew Research Center. http://www.pewinternet.org/2015/10/29/technology-device-ownership-2015/

Aher, R. S., Mayekar, A., & Malik, N. (2017). Mobile technology in library service. Sansmaran Research Journal, 7 (2), 63–69.

Google Scholar  

Beddall-Hill, N. L., Jabbar, A., & Al Shehri, S. (2011). Social mobile devices as tools for qualitative research in education: iPhones and iPads in ethnography, interviewing, and design-based research. Journal of the Research Center of Educational Technology, 7 (1), 67–90.

Bell, A., Headrick Taylor, K., Riesland, E., & Hays, M. (2019). Learning to see the familiar: Technological assemblages in a higher education (non)classroom setting. British Journal of Educational Technology, 50 (4), 1573–1588. https://doi.org/10.1111/bjet.12800

Bowen, K., & Pistilli, M. D. (2012). Student preferences for mobile app usage. Edu-cause Research Bulletin.

Breeding, M. (2019). Investing in enhanced digital experience. Computers in Libraries, 39 (8), 9–11.

Burch, Z. A., & Mohammed, S. (2019). Exploring Faculty Perceptions about Classroom Technology Integration and Acceptance: A literature Review. The Richard W. Riley College of Education and Leadership Publications. 196. https://scholarworks.waldenu.edu/cel_pubs/196

Canuel, R., & Chrichton, C. (2015). Leveraging apps for research and learning: A survey of Canadian academic libraries. Library Hi Tech, 33 (1), 2–14. https://doi.org/10.1108/LHT-12-2014-0115

Canuel, R., MacKenzie, E., Senior, A., & Torabi, N. (2016). Apps for academic success: Developing digital literacy and awareness to increase usage. Education for Information, 33 (3), 1–18. https://doi.org/10.3233/EFI-160085

Carlos, A. (2012). Research on the go: Mobile tools for conducting research. The Reference Librarian, 53 (4), 433–440.

Carter, A., Liddle, J., Hall, W., & Chenery, H. (2015). Mobile phones in research and treatment: Ethical guidelines and future directions. JMIR Mhealth and UHealth, 3 (4), 1.

Chen, G. (2011). Mobile research: Benefits, applications and outlooks. Internationalization, Design and Global Development, 1 , 11–16.

Chen, B., Sivo, S., Seilhamer, R., Sugar, A., & Mao, J. (2013). User acceptance of mobile technology: A campus-wide implementation of blackboard’s MobileTM learn application. Journal of Educational Computing Research, 49 (3), 327–343. https://doi.org/10.2190/EC.49.3.c

Chen, B., Seilhamer, R., Bennet, L., & Bauer, S. (2015). Students’ mobile learning practices in higher education: A multi-year study. Educause Review. http://er.educause.edu/articles/2015/6/students-mobile-learning-practices-in-higher-education-a-multiyear-study

Compton, H., & Burke, D. (2018). The use of mobile learning in higher education: A systematic review. Computers & Education, 123 , 53–64. https://doi.org/10.1016/j.compedu.2018.04.007

Davidson, J., Paulus, T., & Jackson, K. (2016). Speculating on the future of digital tools for qualitative research. Qualitative Inquiry, 22 (7), 606–610.

EDUCAUSE (2019). Horizon Report Preview: 2019 Higher Education Edition . https://library.educause.edu/-/media/files/library/2019/2/2019horizonreportpreview.pdf

Fosnacht, K., Sarraf, S., Howe, E., & Peck, L. K. (2017). How important are high response rates for college surveys? The Review of Higher Education, 40 (2), 245–265. https://doi.org/10.1353/rhe.2017.0003

Galanek, J. D., Gierdowski, D. C., & Brooks, D. C. (2018). ECAR study of undergraduate students and information technology, 2018 . Research report. Louisville, CO: ECAR. https://library.educause.edu/-/media/files/library/2018/10/studentitstudy2018.pdf?la=en&hash=C590C1F6C62B77792711BFAC1F642254A5618590

Galanek, J. D., & Gierdowski, D. C. (2019). ECAR Study of Faculty and Information Technology, 2019. Research report. Louisville, CO: ECAR. https://www.educause.edu/-/media/files/library/2019/12/facultystudy2019.pdf?la=en&hash=F0F8AF83BD870239A7E5CD285FBDBE157AA51135

Gierdowski, D. C., Brooks, D. C, & Galanek, J. D. (2020) EDUCAUSE 2020 Student Technology Report: Supporting the Whole Student. Research report. CO: ECAR. https://www.educause.edu/ecar/research-publications/student-technology-report-supporting-the-whole-student/2020/technology-use-and-environmental-preferences

Garcia, B., Welford, J., & Smith, B. (2016). Using a smartphone app in qualitative research: The good, the bad and the ugly. Qualitative Research, 16 (5), 508–525.

Gikas, J., & Grant, M. M. (2013). Mobile computing devices in higher education: Student perspectives on learning with cellphones, smartphones & social media. The Internet and Higher Education, 19 , 18–26. https://doi.org/10.1016/j.iheduc.2013.06.002

Hahn, J. (2014). Undergraduate research support with optical character recognition apps. Reference Services Review , 42 (2), 336-350.

Hanbridge, A. S., Tin, T., & Sanderson, N. (2018). Information literacy skills on the go: Mobile learning innovation. Journal of Information Literacy, 12 (1), 118–136.

Hennig, N. (2014). Apps for librarians: Using the best mobile technology to educate, create and engage. Libraries Unlimited. https://ebookcentral-proquest-com.ezproxy.waikato.ac.nz/lib/waikato/detail.action?docID=1813321

Hinze, A., Vanderschantz, N., Timpany, C., Cunningham, S. J., Saravani, S. J., & Wilkinson, C. (2017a). Use of mobile apps for teaching and research . (Working Paper Series 01/2017a). Department of Computer Science, The University of Waikato.

Hinze, A., Vanderschantz, N., Timpany, C., Saravani, S. J., Cunningham, S. J., & Wilkinson, C. (2017b). Use of mobile apps for teaching and research–implications for digital literacy. In International Conference on Asian Digital Libraries (pp. 173–184). Springer.

Kim, B. (2013). Responsive web design, discoverability, and mobile challenge. The library mobile experience: Practices and user expectations. Library Technology Reports, 49 (6), 29–39. https://doi.org/10.5860/ltr.49n6

Lai, K.-W., & Smith, L. (2018). Socio-demographic factors relating to perception and use of mobile technologies in tertiary teaching. British Journal of Educational Technology, 49 , 492–504. https://doi.org/10.1111/bjet.12544

MacNeill, F. (2015). Approaching apps for learning, teaching and research. In: A. Middleton (Ed.), Smart learning: Teaching and learning with smartphones and tablets in post compulsory education (pp. 238–264).

McGeeney, K. (2015). What we learned about surveying with mobile apps. Report by Pew Research Center. http://www.pewresearch.org/fact-tank/2015/04/02/what-we-learned-about-surveying-with-mobile-apps/

Morris, N. P., Lambe, J., Ciccone, J., & Swinnerton, B. (2016). Mobile technology: Students perceived benefits of apps for learning neuroanatomy. Journal of Computer Assisted Learning, 32 , 430–442. https://doi.org/10.1111/jcal.12144

Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment & Evaluation in Higher Education, 33 (3), 301–314. https://doi.org/10.1080/02602930701293231

Pedro, L., Barbosa, C., & Santos,C. (2018). A critical review of mobile learning integration in formal educational contexts. International Journal of Educational Technology in Higher Education, 15 (10), 1. https://doi.org/10.1186/s41239-018-0091-4

Perrin, A. (2021). Mobile Technology and Home Broadband 2021. Report by Pew Research Center. https://www.pewresearch.org/internet/2021/06/03/mobile-technology-and-home-broadband-2021/

Raento, M., Oulasvirta, A., & Eagle, N. (2009). Smartphones: An emerging tool for so-cial scientists. Sociological Methods & Research, 37 (3), 426–454.

Saragossi, J., Kasten, K., & Costello, L. (2018). Mobile applications in academic libraries. Library Faculty Publications, 21 . https://commons.library.stonybrook.edu/library_articles/21

Schepman, A., Rodway, P., Beattie, C., & Lambert, J. (2012). An observational study of undergraduate students’ adoption of (mobile) note-taking software. Computers in Human Behavior, 28 , 308–317. https://doi.org/10.1016/j.chb.2011.09.014

Sensor Tower. (2019). Global app revenue grew 23% in 2018 to more than $71 billion on iOS and Google Play . https://sensortower.com/blog/app-revenue-and-downloads-2018

Shraim, K., & Crompton, H. (2015). Perceptions of using smart mobile devices in higher education teaching: A case study from palestine. Contemporary Educational Technology, 6 (4), 301–318.

Shuk Han Wai, I., Sze Yan Ng, S., Chiu, D. K. W., Ho, K. K. W., & Lo, P. (2016). Exploring undergraduate students’ usage pattern of mobile apps for education. Journal of Librarianship and Information Science. https://doi.org/10.1177/0961000616662699

Singh Negi, D. (2014). Using mobile technologies in libraries and information centers. Library Hi Tech News, 31 (5), 14–16.

Singh, G., & Hardaker, G. (2014). Barriers and enablers to adoption and diffusion of eLearning: A systematic review of the literature–a need for an integrative approach. Education+ Training, 56 (2/3), 105–121.

Statista. (2019). Most popular Apple App Store categories in May 2019, by share of available apps . https://www.statista.com/statistics/270291/popular-categories-in-the-app-store/

Tidal, J. (2017). One site to rule them all, redux: The second round of usability testing of a responsively designed website. Journal of Web Librarianship, 11 (1), 16–34. https://doi.org/10.1080/19322909.2016.1243458

van Arnhem, J-P. (2015). Apps and gear for ethnographic field research. The Charleston Advisor, 17 (2), 58-64.

Van Mol, C. (2017). Improving web survey efficiency: The impact of an extra reminder and reminder content on web survey response. International Journal of Social Research Methodology, 20 (4), 317–327. https://doi.org/10.1080/13645579.2016.1185255

Wong, S. H. R. (2012). Which platform do our users prefer: Website of mobile app? Reference Services Review, 40 (1), 103-115.

Download references

Open Access funding enabled and organized by CAUL and its Member Institutions.

Author information

Authors and affiliations.

Department of Software Engineering, University of Waikato, Hamilton, New Zealand

Annika Hinze & Sally Jo Cunningham

Department of Design, University of Waikato, Hamilton, New Zealand

Nicholas Vanderschantz & Claire Timpany

University Library, University of Waikato, Hamilton, New Zealand

Sarah-Jane Saravani & Clive Wilkinson

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Annika Hinze .

Ethics declarations

Conflict of interest, additional information, publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Hinze, A., Vanderschantz, N., Timpany, C. et al. A Study of Mobile App Use for Teaching and Research in Higher Education. Tech Know Learn 28 , 1271–1299 (2023). https://doi.org/10.1007/s10758-022-09599-6

Download citation

Accepted : 10 March 2022

Published : 05 June 2022

Issue Date : September 2023

DOI : https://doi.org/10.1007/s10758-022-09599-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mobile apps
• Academic survey
• Tertiary teaching and research
• Information behavior
• Twenty-first century abilities
• Find a journal
• Publish with us
• Track your research

This paper is in the following e-collection/theme issue:

Published on 1.4.2024 in Vol 8 (2024)

Development and Evaluation of a Digital App for Patient Self-Management of Opioid Use Disorder: Usability, Acceptability, and Utility Study

Authors of this article:

Original Paper

• Van Lewis King Jr 1 , MD   ; 
• Gregg Siegel 2 , MSc   ; 
• Henry Richard Priesmeyer 3 , PhD   ; 
• Leslie H Siegel 2 , MFA   ; 
• Jennifer S Potter 1 , PhD  

1 Department of Psychiatry and Behavioral Sciences, University of Texas Health Science Center San Antonio, San Antonio, TX, United States

2 Biomedical Development Corporation, San Antonio, TX, United States

3 Department of Management and Marketing, St. Mary's University, San Antonio, TX, United States

Corresponding Author:

Van Lewis King Jr, MD

Department of Psychiatry and Behavioral Sciences

University of Texas Health Science Center San Antonio

5109 Medical Drive

San Antonio, TX, 78229

United States

Phone: 1 210 450 8058

Fax:1 210 450 2104

Email: [email protected]

Background: Self-management of opioid use disorder (OUD) is an important component of treatment. Many patients receiving opioid agonist treatment in methadone maintenance treatment settings benefit from counseling treatments to help them improve their recovery skills but have insufficient access to these treatments between clinic appointments. In addition, many addiction medicine clinicians treating patients with OUD in a general medical clinic setting do not have consistent access to counseling referrals for their patients. This can lead to decreases in both treatment retention and overall progress in the patient’s recovery from substance misuse. Digital apps may help to bridge this gap by coaching, supporting, and reinforcing behavioral change that is initiated and directed by their psychosocial and medical providers.

Objective: This study aimed to conduct an acceptability, usability, and utility pilot study of the KIOS app to address these clinical needs.

Methods: We developed a unique, patient-centered computational software system (KIOS; Biomedical Development Corporation) to assist in managing OUD in an outpatient, methadone maintenance clinic setting. KIOS tracks interacting self-reported symptoms (craving, depressed mood, anxiety, irritability, pain, agitation or restlessness, difficulty sleeping, absenteeism, difficulty with usual activities, and conflicts with others) to determine changes in both the trajectory and severity of symptom patterns over time. KIOS then applies a proprietary algorithm to assess the individual’s patterns of symptom interaction in accordance with models previously established by OUD experts. After this analysis, KIOS provides specific behavioral advice addressing the individual’s changing trajectory of symptoms to help the person self-manage their symptoms. The KIOS software also provides analytics on the self-reported data that can be used by patients, clinicians, and researchers to track outcomes.

Results: In a 4-week acceptability, usability (mean System Usability Scale-Modified score 89.5, SD 9.2, maximum of 10.0), and utility (mean KIOS utility questionnaire score 6.32, SD 0.25, maximum of 7.0) pilot study of 15 methadone-maintained participants with OUD, user experience, usability, and software-generated advice received high and positive assessment scores. The KIOS clinical variables closely correlated with craving self-report measures. Therefore, managing these variables with advice generated by the KIOS software could have an impact on craving and ultimately substance use.

Conclusions: KIOS tracks key clinical variables and generates advice specifically relevant to the patient’s current and changing clinical state. Patients in this pilot study assigned high positive values to the KIOS user experience, ease of use, and the appropriateness, relevance, and usefulness of the specific behavioral guidance they received to match their evolving experiences. KIOS may therefore be useful to augment in-person treatment of opioid agonist patients and help fill treatment gaps that currently exist in the continuum of care. A National Institute on Drug Abuse–funded randomized controlled trial of KIOS to augment in-person treatment of patients with OUD is currently being conducted.

Introduction

The United States is experiencing an opioid epidemic [ 1 ]. In 2021, 106,699 people died from overdoses involving opioids, including both prescription and illicit opioids, and the number of deaths continues to increase [ 1 ]. Severe opioid use disorder (OUD) is a lifelong and costly illness affecting millions of people worldwide. The socioeconomic impact of OUD affects every aspect of a person’s life; repercussions from long-term opioid use often include legal problems, damage to personal relationships, and significant morbidity and mortality [ 2 ].

One of the most effective treatments for severe OUD typically involves opioid agonist treatment [ 3 , 4 ]. While the benefits of medication treatment often occur rapidly, positive behavioral changes for those with more severe OUD can take months and years to develop. Patients receive professional treatment in the clinic, but the struggle with addiction occurs largely in the setting of their everyday lives. OUD treatment often involves periods of exacerbation and remission, and the vulnerability to recurrent drug use remains a life-long risk for many patients. Treatment adherence is often intermittent, and the potential for a recurrence of substance use disorder symptoms carries with it the chronic risk of overdose, trauma, suicide, and infectious diseases [ 2 , 5 ].

Self-management of OUD is a critical component of recovery. Behavioral and supportive therapy used in conjunction with medication is often helpful in changing deeply embedded behaviors that can lead to recurrent drug use [ 5 , 6 ]. However, patients often face a shortage of mental health services, which limits access to care, education, and counseling and impedes their ability to develop effective patient self-management behaviors [ 7 ]. Ongoing coaching and feedback are important for recovery for many patients but difficult and expensive to implement, and generally inaccessible during the time of greatest need.

Digital therapeutic software apps are intended to help address this challenge with a variety of approaches being used, and many patients are open to these treatment enhancements [ 8 ]. Recognizing the potential of this research and the benefits of evaluating multiple treatment approaches, the National Institutes of Health has increasingly supported digital health behavior research during the past 15-20 years [ 9 , 10 ]. The National Institute on Drug Abuse Clinical Trials Network has conducted several digital health studies examining digital therapeutics for treatment and other technologies for screening, assessment, and a range of other uses [ 11 ].

There are growing numbers of increasingly sophisticated apps to address various substance use problems [ 12 - 16 ]. These apps use diverse approaches [ 17 , 18 ], and there is evidence of benefits and cost-effectiveness associated with their use such as improved retention and reduced drug use [ 18 - 20 ].

This study examines the usability, acceptability, and utility of KIOS, a patient-centered computational software system [ 21 ]. The KIOS app uses a mobile technology platform to provide daily symptom monitoring and on-demand, individualized feedback. Feedback is based on changes in key treatment variables between successive self-reports to teach and reinforce healthy practices, foster self-management, and promote adherence to treatment plans for patients enrolled in methadone maintenance treatment for OUD. Since engagement with the digital device is key to determining possible efficacy, this study was conducted to make a preliminary evaluation of the KIOS software and client interface in preparation for a randomized controlled trial in a treatment clinic setting.

This study used a mixed model design to demonstrate technical feasibility; patient engagement; and the acceptability, usability, and utility of KIOS with individuals receiving methadone maintenance for OUD.

Ethical Considerations

The focus group protocol was approved by BioMed IRB (approval number 18-1-100-OUD). Participants provided informed written consent and were paid for their participation. In addition, the KIOS use study protocol was approved by BioMed IRB and participants provided signed informed consent and were enrolled in a 4-week, single-group, pre-post evaluation of the KIOS app. Participants were paid US $75 after completing orientation and training. They were paid an additional US $75 for participating in a debriefing or focus group. All patient responses were stored in a confidential fashion in a password-protected computer database and only available to approved study personnel. All data were deidentified prior to analysis.

Developing the User Interface

Contextual design [ 22 ], an industry-standard, user-centered methodology, was used to design the user experience. Individuals receiving methadone maintenance were recruited for focus groups at a private treatment clinic (CleanSlate Addiction Treatment Center, n=5) and the local public treatment clinic (The Center for Healthcare Services, n=9). Participants provided informed written consent for their participation. Feedback from participants was solicited about the perceived usefulness of the tool, functionality, benefits, and incentives for using the software.

Based on the focus group feedback, a functional user interface (UI) was created. The web-based KIOS UI includes assessment screens for each of the clinical variables (craving, depressed mood, anxiety, irritability, pain, agitation or restlessness, difficulty sleeping, absenteeism, difficulty with usual activities, and conflicts with others; each rated on a 1-7 scale from none to severe). The determination of the clinical variables is described in the Results section. Inside KIOS, the patient rates the severity of these 10 symptoms and logs self-reported opioid use, alcohol use, and recreational drug use in an important behavior checklist. Participants also reported regular exercise, avoidance of triggers or high-risk situations, and sleep habits (not reported here). The KIOS app analyzes the changes in symptoms during the time between patient self-reports and then delivers advice specific to the patient’s evolving symptoms. On the advice screen of the app, KIOS contextually suggests behaviors that are associated with improving self-report variables, including reaching out to support people or treatment providers, if indicated. In these circumstances, the app encourages the integrated use of comprehensive treatment resources (eg, for increasing depression, patients are directed to consult a medical provider). The results of both the symptom changes and the reporting on important behaviors could also be reviewed by the patient’s individual counselor. This offers the counselor access to real-time data over time on both symptoms and behaviors. It offers the ability to reinforce behavioral treatment planning and maximize the limited counseling time available at each appointment. KIOS also contains a home and menu screen, 6 graphs for tracking symptoms over time, a primary advice screen, a second supplemental advice screen if patients want additional advice, a calendar-based journal, and help screens for technical support, feedback, and emergency contacts for help or crisis situations.

KIOS Use Study Description

The primary outcome was user satisfaction as measured by acceptability, usability, and utility. Secondary outcomes included self-reporting of drug and alcohol use. The study was conducted exclusively on a web-based interface and via telephone. Participants were recruited from Community Medical Services, a private opioid treatment organization. Staff at Community Medical Services provided potential participants with study information and individuals contacted study staff if they wanted to participate.

Inclusion criteria were (1) interest in study participation; (2) 18 years or older; (3) receiving methadone maintenance for OUD for ≥4 weeks; (4) ability to access KIOS via computer, smartphone, or tablet; and (5) no unmanaged major psychiatric illness or suicidality.

Participants attended a web-based KIOS training session. They were asked to complete KIOS assessments on a web-based interface at least 3 times per week but no more than once daily. KIOS could still be accessed by participants as often as desired to review advice and graphs or access the journal. All assessments were logged on the KIOS server to tabulate the frequency of use.

After the trial, participants completed an 18-question KIOS survey. Participants also completed the System Usability Scale (SUS)-Modified, a single-factor 10-item self-report scale that was used to evaluate participants’ subjective experience using KIOS [ 23 , 24 ].

User data were logged on the KIOS server for the 10 variables from each assessment. At the end of the study, a final web-based debriefing focus group was conducted to gather user feedback not otherwise collected during the study. Participants described their experience in the study and provided their impressions, suggestions, and critiques of KIOS in an open dialogue.

Means, medians, and SDs were calculated for usage, SUS, and patient questionnaire data. KIOS variable scores were averaged over each week for each participant to account for differing numbers of assessments. Calculations were similarly made for an overall KIOS index score (sum of all variables except craving) and for emotional (depressed mood, anxiety, and irritability), physical (pain, agitation or restlessness, and difficulty sleeping), and social (absenteeism, difficulty with usual activities, and conflicts with others) subscales. The craving variable was kept separate due to a consistent body of literature linking it to substance use outcomes [ 25 - 27 ]. The changes in mean values for each week were calculated relative to baseline measures, and 2-tailed Student t tests were conducted for significance between baseline and the weekly means. A Pearson correlation was performed by comparing the KIOS index score to the craving scores averaged over the study period.

Identifying Key Treatment Variables

Managing recovery from OUD is a complex process characterized by constantly changing intra- and interpersonal circumstances. It was essential to identify a limited number of relevant clinical variables that would both effectively reflect an individual’s subjective behavioral state as well as provide sufficient information to permit the assignment of appropriate behavioral advice. Deidentified data from large-scale randomized OUD clinical trials (the POATS trial: n=653 and the X:BOT trial: n=570) [ 28 , 29 ] were analyzed to provide relevant clinical variables. The KIOS software requires clinical variables that change frequently to operate optimally. Therefore, data obtained from instruments in these studies were analyzed to determine how frequently variables changed from one assessment to the next.

An expert panel ( Table 1 ) selected a subset of 32 variables that were most relevant to OUD recovery. The members of the panel were selected for their expertise in behavioral interventions in OUD and to impart a multicenter perspective to the study. To achieve panel consensus, a modified Delphi process was used [ 30 ]. A web-based survey tool and web-based meetings were used to expedite the process. The Delphi process was iterated until consensus was achieved. The decision rule for the inclusion of variables into the system called for the variable to be selected by a majority of the panel members.

a SUD: substance use disorder.

b OUD: opioid use disorder.

c CBT: cognitive behavioral therapy.

Independently, each panel member selected variables with the criteria that a change in the variable is important in determining the patient’s current clinical state. If consensus was not achieved, the panel was queried until the decision rule was met. The variables proposed by the expert panel were then evaluated in a correlation matrix to eliminate redundant variables. Ten individual variables (craving, depressed mood, anxiety, irritability, pain, absenteeism, agitation or restlessness, difficulty sleeping, difficulty with usual activities, and conflicts with others) were selected to capture a person’s clinical state most comprehensively while minimizing the number of questions required of the patient at any assessment.

Next, the panelists proposed combinations of 2 or more variables that provided the most valuable information for describing the current state of the patient and that would be particularly sensitive to identifying transitional worsening or improvement in OUD. This grouping of variables created the underlying structure that mapped the patient trajectory and enabled the development of interventions. Reports were generated by the KIOS software describing all the possible changes for the selected set of interacting variables. The panel supplemented these descriptions with interpretations and recommended interventions.

Appropriate behavioral interventions incorporating cognitive behavioral therapy (CBT) and mindfulness, using the Behavior Change Technique Taxonomy [ 31 ] as a reference, were applied to each patient state. These change techniques have been shown effective in multiple studies [ 32 ] ( Table 2 ). All recommendations were phrased in patient-friendly language and subjected to a rubric specifying reading level and text length, and that the advice is consistent with a CBT approach. In a previous study, the research team successfully followed a similar approach to chart trajectories and develop self-management interventions for bipolar disorder [ 21 ].

Participants

In total, 19 individuals signed informed consent. Three did not complete orientation. One stopped participating after 11 days and was considered lost to follow-up. The median time in methadone treatment at the beginning of the study was 9 months: 4 patients had been in treatment for ≤6 months, 6 patients had been in treatment for 6-12 months, and 5 had been in treatment for >12 months. Participants (N=19) were 47% (n=9) White (non-Hispanic), 21% (n=4) White (Hispanic), 11% (n=2) more than 1 race, 11% (n=2) American Indian or Alaska Native, 5% (n=1) Black or African American, and 5% (n=1) Asian. The mean age of all participants was 34.4 (SD 7.3) years, and 63% (n=12) were women.

KIOS Use and Acceptability

In total, 15 of 16 participants who logged in to KIOS at least once finished the trial. Those 15 participants completed 191 total assessments, averaging 12.7 (SD 3.8) per user (median 13, IQR 11-14). During the poststudy debriefings, several participants requested continued access to KIOS after the trial ended without further compensation; 6 participants used KIOS poststudy and 1 participant used the app for over 10 months.

The final debriefing focus group generated many comments that described the individualized experience of using the app. All participants who attended the group stated it was helpful and worth using. For example, focus group attendees noted that KIOS gave timely and responsive advice, helped with adherence to treatment goals, and promoted reflection and motivation. Some commented on how natural the advice seemed, almost like a counselor was responding in real time.

Some participants mentioned features that could improve KIOS such as a customized reminder system to specify the time of day to use the app, features to connect directly with counseling staff or peer support, gamified content (such as daily challenges, more content that included pictures), positive reinforcement (such as digital trophies, badges, or monetary rewards), and expanded and more specific description of exercise or meditation practices beyond those currently included in the app.

KIOS Usability

The mean SUS-Modified score was 89.5 (SD 9.2; median 92.5, IQR 85-95). Higher scores indicate greater usability. The psychometric properties of the SUS have been validated and replicated, and the score for KIOS falls slightly below the top score, “best imaginable=90.9” and above “excellent=85.5” on an adjective rating of the SUS [ 23 , 24 ] ( Table 3 ).

KIOS Utility

The KIOS questionnaire was developed by the research team to evaluate the utility of the KIOS app. The mean response to all KIOS questionnaire data was 6.32 out of a maximum of 7.0 (SD 0.25), indicating an overall strong agreement with most statements. The highest rated statement was “I see benefits using KIOS,” which averaged 6.73 out of 7 (SD 0.46); each respondent rated this item as either 6 or 7. The lowest rated item was “KIOS helped identify some personal triggers,” which was still generally well agreed upon and rated 5.73 (SD 1.62; Table 4 ).

KIOS Assessment Data

The Healthy Behavior Checklist allows patients to check yes or no for engaging in the behavior at each assessment. The behaviors include opioid use, recreational drug use, alcohol use, exercising, good sleep habits, and avoiding triggers and high-risk situations. Only 1 participant self-reported 1 occurrence of unprescribed opioid use. This occurred at the baseline assessment, and no other illicit opioid use was reported by any participant during the study. Four participants reported 18 occurrences of alcohol use and 3 others reported 24 instances of recreational drug use (nonspecified); no participant reported both alcohol and recreational drug use.

Figures 1 and 2 demonstrate how KIOS data can be used to track patient reporting over time. In Figure 1 , the KIOS score and craving score were highly correlated, indicating that the symptom-reporting scales appear to be closely related to self-reported craving scores over time (Pearson r =0.550; P <.001). Two KIOS subscales were significantly lower at the end of the trial. Emotional (week 4; P =.01) and social (week 4; P =.02) subscales were significantly lower compared to the baseline ( Figure 2 ).

Principal Findings

The KIOS app is an adjunctive tool to OUD treatment that gives patients 24/7 access to behavioral intervention strategies and supportive and reinforcing behavioral guidance that potentially could augment and improve response to routine clinic-based counseling interventions. In this study, participants rated the primary outcomes of acceptability, usability, and utility very highly. The scores on the SUS-modified were in the excellent range. The mean response to all KIOS questionnaire data was 6.32 out of a maximum of 7.0 (SD 0.25), indicating an overall strong agreement. Participants used KIOS on average about 3 times weekly, which is a very good indication of sufficient use of the app to generate relevant advice.

KIOS Acceptability

Participants on average used KIOS 3 times per week, which is the minimum requested in the study. This allowed sufficient time for relevant behavioral change between self-reports to generate pertinent recovery advice to the user. However, this may indicate a need to make the KIOS app more engaging, since participants could use the app daily if desired. Focusing the content on participants earlier in OUD treatment who may want more recovery advice or improving the experience by augmenting personalization could make the app more engaging [ 33 , 34 ]. Revisions to the app will include more explicit recommendations to continue in OUD treatment and more varied suggestions about mindfulness-oriented and positive self-care activities. Patients commented favorably on the individualized and specific nature of the advice and how this motivated and kept them on track in recovery. Some even commented that it seemed a counselor was responding to their self-report assessments in real time. This sense of copresence, of someone concurrently participating in the intervention with a psychological connection to the app user, was associated with improved satisfaction and efficacy in studies of other software apps [ 35 , 36 ].

The SUS-Modified is a well-validated instrument [ 23 , 24 ]. KIOS was rated as very usable for these patients who were recruited from a community-based methadone maintenance treatment program. One item stood out as less positive: how cumbersome the app was to use. Since ease of use is clearly important, revisions to the app will involve attention to this aspect of use (eg, ease of moving between the home screen and different app functions). The revised KIOS app will be used on a smartphone platform to improve this aspect of usability.

Utility was measured by a questionnaire specifically developed to evaluate the KIOS app. Participants strongly agreed on most questions related to usefulness, applicability, and relevance to recovery. Items that were less strongly endorsed were related to identifying personal triggers and improved health status. Since about one-third of the participants had been in treatment over 12 months, it is possible that identifying personal triggers was less of a treatment concern, and therefore, the mean score for this item was somewhat lower. In addition, since the study was only 4 weeks in duration, there was not much time to notice changes in general health status. Self-report of drug and alcohol use was modest in this sample, where only a minority of the participants was in treatment for less than 6 months. Much of the content of the app is aimed at changes in symptoms related to substance use problems; yet, participants who had minimal drug use problems and few mental health concerns rated the experience as helpful and relevant to their recovery. The randomized controlled study of KIOS will have a duration of 6 months and only focus on patients who are early in the treatment process. This will result in greater changes in drug use behavior and a longer period of time to capture potential changes in personal triggers and general health status. Revisions to the app will include more focus on positive behavioral changes to improve patient self-management and social support for recovery.

KIOS Assessment Self-Reports

The scores derived from the KIOS self-report generated potentially useful data for both the KIOS user and the treatment team. As expected, the KIOS score and craving score were highly correlated. The correlation between these 2 measures suggests that the assessment variables, which collectively form the KIOS score capture relevant emotional, social, and physical phenomena that correspond with self-reported craving. Figures 1 and 2 demonstrate how KIOS data can usefully track patient reporting over time. These results could also be reviewed by the patient’s counselor to reinforce behavioral treatment planning and maximize the limited counseling time available at each appointment to assist the patient with this aspect of their care. The KIOS journal feature is available to help the patient record and organize pertinent thoughts and reflections, and the behavioral checklist helps to organize and track drug use and positive behaviors associated with improved recovery outcomes.

KIOS advice is designed to track patient self-report items related to OUD recovery over time. The KIOS app modifies the advice given to the user based on the person’s changes in self-report at each assessment. Advice is then directed specifically to the symptoms of most concern to the user. KIOS can draw the patient’s attention to triggers and potential problem behaviors with helpful and supportive advice in real time to reinforce recovery activities and goals at times of vulnerability to drug use. It may advise reaching out to support people or treatment providers depending on the type of a variable or increasing intensity of symptom reporting (eg, for increasing depression and insomnia consulting a medical provider), further encouraging integrated use of comprehensive treatment resources. It gives specific, evidence-informed suggestions for various symptoms including sleep hygiene, stress management, avoiding triggers, and pain reduction behaviors.

Expanded access to digital behavioral interventions has the potential to bridge a significant treatment gap due to the lack of counseling resources in many OUD treatment settings. However, there are very few digital behavioral apps that specifically address OUD [ 13 , 17 , 37 , 38 ]. Due to the nearly uniformly serious nature of this disorder, it is unlikely that a digital app by itself would be potent enough to help a person manage this disorder without professional treatment participation. The available apps focus primarily on connecting users to treatment services and peer support, making available CBT educational modules, using self-report check-ins, and using contingency management interventions. Their use has demonstrated significant improvements in some treatment outcomes [ 39 ]. For example, studies of reSET-O showed efficacy regarding opioid use or abstinence and in reinforcing and increasing treatment activity frequency in controlled trials [ 40 ]. The web-based CBT4CBT app improved retention and reduced drug use when combined with office-based treatment as usual [ 41 ]. It is one of the few digital interventions that has shown improved efficacy compared to in-person CBT in randomized controlled trials and demonstrates the potential power of these types of treatment augmentations [ 41 , 42 ]. The results of a randomized controlled trial of the A-CHESS digital app for use as an adjunct treatment in OUD did not demonstrate significant benefits compared to the control condition [ 16 , 43 ]. However, some results indicated that for specific subpopulations there could be benefits.

Other apps use a therapeutic relational approach using a conversational agent, such as those using a chatbot platform [ 12 , 44 ]. Some of these interventions have demonstrated efficacy [ 45 ]. They are currently used to help patients with a variety of mental health issues, although few have an evidence base [ 13 , 45 - 47 ]. The use of the W-SUDs modification of the Woebot app [ 12 , 13 ] has been associated with self-reported reductions in drug use and urges to use drugs and improvement in anxiety in users with substance use concerns [ 13 ]. The Woebot gives the impression of a therapeutic coach in the conversation generated by the chatbot. There is evidence from prior studies that some users are more likely to disclose information [ 48 ] to nonhuman apps, and that a strong therapeutic alliance can form [ 49 - 51 ]. Although not measured in this study, some participants commented on how KIOS responses seemed like real-time exchanges with a therapist. Future studies of KIOS should examine this aspect of the UI.

The study limitations include the short study duration in this trial to primarily test the acceptability, utility, and usability of the software. There was no control group, and it was not compared to another treatment condition or substance use treatment app to determine the specific benefits of KIOS compared to other apps or interventions for persons with substance use problems. The small sample size limits generalizability in several ways: to other opioid agonist treatment clinics or office-based OUD treatment, other patient samples that may be less interested in using digital apps, and in the ability to detect statistical differences between study variables. The patient focus group comments reflected the views of persons who were interested in using the app and potentially predisposed to having a positive user experience. KIOS relies on patient self-reporting to generate advice, so if a patient did not report accurate information, then they would not get accurate advice. This could be a problem if the patient’s counselor had access to the patient’s data. For example, if the patient was not ready to reveal their substance use, they might not give an accurate self-report. This should also be examined in future studies of the app.

Conclusions

The KIOS app offers a different approach to the digital app for augmenting outpatient treatment for OUD. Potentially, the advice generated from the app may be more specific to the user because self-reported symptom clusters are used to identify appropriate behavioral interventions selected from evidence-based literature. It also has the potential advantage of using the therapeutic relational coaching approach that many users find more engaging.

Current plans for the development of KIOS include adding varied advice features, incorporating and reinforcing more positive recovery behaviors (eg, positive social support), and pursuing Food and Drug Administration approval as a prescription medical device. Although KIOS was studied in methadone maintenance clinics, it may be a very helpful tool in office-based buprenorphine addiction medicine treatment settings that often also have limited counseling availability. A National Institute on Drug Abuse–funded randomized controlled trial of KIOS is currently being conducted to evaluate its efficacy.

Acknowledgments

The authors thank the patients and staff of CleanSlate, The Center for Health Care Services, and Community Medical Services in San Antonio, TX, and Cedar Park, TX, for their assistance and participation in the study and to the consultants on the panel who assisted in developing the KIOS software app. The authors gratefully acknowledge Charles L Bowen, MD, the principal investigator on the grant that supported this work. Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under award number R43MH119672 (principal investigator: Charles Bowden). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Data Availability

The data sets generated and analyzed during this study are not publicly available due to the proprietary nature of the device. Reasonable requests from the KIOS inventor, HRP, should be addressed to him specifically.

Conflicts of Interest

GS and LHS work for or own stocks or stock options of the company that developed or owned the app, product evaluated, and discussed in the paper. HRP, GS, and LHS were involved in the development of the innovation, software, and app they have been evaluating. VLK and JSP have no financial conflicts of interest.

• Centers for Disease Control and Prevention. URL: https://www.cdc.gov [accessed 2023-10-18]
• American Psychiatric Publishing. DSM-5-TR. 5th Edition. Washington, D.C. American Psychiatric Publishing; 2022.
• Ayanga D, Shorter D, Kosten TR. Update on pharmacotherapy for treatment of opioid use disorder. Expert Opin Pharmacother. 2016;17(17):2307-2318. [ CrossRef ] [ Medline ]
• Substance Abuse and Mental Health Services Administration. URL: https://www.samhsa.gov/find-help/national-helpline [accessed 2023-10-18]
• Rice D, Corace K, Wolfe D, Esmaeilisaraji L, Michaud A, Grima A, et al. Evaluating comparative effectiveness of psychosocial interventions adjunctive to opioid agonist therapy for opioid use disorder: a systematic review with network meta-analyses. PLoS One. 2020;15(12):e0244401. [ FREE Full text ] [ CrossRef ] [ Medline ]
• King VL, Brooner RK. Improving treatment engagement in opioid-dependent outpatients with a motivated stepped-care adaptive treatment model. Jt Comm J Qual Patient Saf. 2008;34(4):209-213. [ CrossRef ] [ Medline ]
• Behavioral health workforce projections 2017-2030. Human Resources and Services Administration. URL: https:/​/bhw.​hrsa.gov/​sites/​default/​files/​bureau-health-workforce/​data-research/​bh-workforce-projections-fact-sheet.​pdf [accessed 2023-10-08]
• Langdon KJ, Scherzer C, Ramsey S, Carey K, Rich J, Ranney ML. Feasibility and acceptability of a digital health intervention to promote engagement in and adherence to medication for opioid use disorder. J Subst Abuse Treat. 2021;131:108538. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Riley WT, Oh A, Aklin WM, Wolff-Hughes DL. National institutes of health support of digital health behavior research. Health Educ Behav. 2019;46(2_suppl):12-19. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Aklin WM, Walton KM, Antkowiak P. Digital therapeutics for substance use disorders: research priorities and clinical validation. Drug Alcohol Depend. 2021;229(Pt A):109120. [ CrossRef ] [ Medline ]
• Marsch LA, Campbell A, Campbell C, Chen CH, Ertin E, Ghitza U, et al. The application of digital health to the assessment and treatment of substance use disorders: the past, current, and future role of the national drug abuse treatment clinical trials network. J Subst Abuse Treat. 2020;112(S):4-11. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health. 2017;4(2):e19. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Prochaska JJ, Vogel EA, Chieng A, Kendra M, Baiocchi M, Pajarito S, et al. A therapeutic relational agent for reducing problematic substance use (Woebot): development and usability study. J Med Internet Res. 2021;23(3):e24850. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pear Therapeutics Inc. US Food and Drug Administration. 2018. URL: https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K173681 [accessed 2024-02-21]
• Velez FF, Colman S, Kauffman L, Ruetsch C, Anastassopoulos K. Real-world reduction in healthcare resource utilization following treatment of opioid use disorder with reSET-O, a novel prescription digital therapeutic. Expert Rev Pharmacoecon Outcomes Res. 2021;21(1):69-76. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gustafson DH, Landucci G, Vjorn OJ, Gicquelais RE, Goldberg SB, Johnston DC, et al. Effects of bundling medication for opioid use disorder with an mhealth intervention targeting addiction: a randomized clinical trial. Am J Psychiatry. 2024;181(2):115-124. [ CrossRef ] [ Medline ]
• Nuamah J, Mehta R, Sasangohar F. Technologies for opioid use disorder management: mobile app search and scoping review. JMIR Mhealth Uhealth. 2020;8(6):e15752. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kiburi SK, Ngarachu E, Tomita A, Paruk S, Chiliza B. Digital interventions for opioid use disorder treatment: a systematic review of randomized controlled trials. J Subst Abuse Treat. 2023;144:108926. [ CrossRef ] [ Medline ]
• Brezing CA, Brixner DI. The rise of prescription digital therapeutics in behavioral health. Adv Ther. 2022;39(12):5301-5306. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hodges J, Waselewski M, Harrington W, Franklin T, Schorling K, Huynh J, et al. Six-month outcomes of the HOPE smartphone application designed to support treatment with medications for opioid use disorder and piloted during an early statewide COVID-19 lockdown. Addict Sci Clin Pract. 2022;17(1):16. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bowden CL, Priesmeyer R, Tohen M, Singh V, Calabrese JR, Ketter T, et al. Development of a patient-centered software system to facilitate effective management of bipolar disorder. Psychopharmacol Bull. 2021;51(2):8-19. [ FREE Full text ] [ Medline ]
• Beyer H, Holtzblatt K. The encyclopedia of human-computer interaction, 2nd Edition. Interaction Design Foundation. URL: https:/​/www.​interaction-design.org/​literature/​book/​the-encyclopedia-of-human-computer-interaction-2nd-ed/​contextual-design [accessed 2024-02-21]
• Bangor A, Kortum PT, Miller JT. An empirical evaluation of the System Usability Scale. Int J Hum-Comput Int. 2008;24(6):574-594. [ CrossRef ]
• Bangor A, Kortum PT, Miller JT. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud. 2009;4(3):114-123. [ FREE Full text ]
• Kakko J, Alho H, Baldacchino A, Molina R, Nava FA, Shaya G. Craving in opioid use disorder: from neurobiology to clinical practice. Front Psychiatry. 2019;10:592. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McHugh RK, Fitzmaurice GM, Carroll KM, Griffin ML, Hill KP, Wasan AD, et al. Assessing craving and its relationship to subsequent prescription opioid use among treatment-seeking prescription opioid dependent patients. Drug Alcohol Depend. 2014;145:121-126. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tsui JI, Anderson BJ, Strong DR, Stein MD. Craving predicts opioid use in opioid-dependent patients initiating buprenorphine treatment: a longitudinal study. Am J Drug Alcohol Abuse. 2014;40(2):163-169. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Weiss RD, Potter JS, Provost SE, Huang Z, Jacobs P, Hasson A, et al. A multi-site, two-phase, Prescription Opioid Addiction Treatment Study (POATS): rationale, design, and methodology. Contemp Clin Trials. 2010;31(2):189-199. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lee JD, Nunes EV, Novo P, Bachrach K, Bailey GL, Bhatt S, et al. Comparative effectiveness of extended-release naltrexone versus buprenorphine-naloxone for opioid relapse prevention (X:BOT): a multicentre, open-label, randomised controlled trial. Lancet. 2018;391(10118):309-318. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sinha IP, Smyth RL, Williamson PR. Using the delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011;8(1):e1000393. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Abraham C, Michie S. A taxonomy of behavior change techniques used in interventions. Health Psychol. 2008;27(3):379-387. [ CrossRef ] [ Medline ]
• Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. 2013;46(1):81-95. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Forbes A, Keleher MR, Venditto M, DiBiasi F. Assessing patient adherence to and engagement with digital interventions for depression in clinical trials: systematic literature review. J Med Internet Res. 2023;25:e43727. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res. 2021;23(3):e24387. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bulu ST. Place presence, social presence, co-presence, and satisfaction in virtual worlds. Comput Educ. 2012;58(1):154-161. [ FREE Full text ] [ CrossRef ]
• LaBrie JW, de Rutte JL, Boyle SC, Tan CN, Earle AM. Leveraging copresence to increase the effectiveness of gamified personalized normative feedback. Addict Behav. 2019;99:106085. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Staiger PK, O'Donnell R, Liknaitzky P, Bush R, Milward J. Mobile apps to reduce tobacco, alcohol, and illicit drug use: systematic review of the first decade. J Med Internet Res. 2020;22(11):e17156. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moore BA, Buono FD, Lloyd DP, Printz DM, Fiellin DA, Barry DT. A randomized clinical trial of the recovery line among methadone treatment patients with ongoing illicit drug use. J Subst Abuse Treat. 2019;97:68-74. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fluetsch N, Whittington MD, Tice JA, Zapata LV, Mendola N, Chapman R, et al. Digital health technologies as an adjunct to medication assisted therapy for opioid use disorder; final evidence report. Institute for Clinical and Economic Review. 2020. URL: https://icer.org/assessment/opioids-digital-apps-2020/#timeline [accessed 2024-02-21]
• Maricich YA, Xiong X, Gerwien R, Kuo A, Velez F, Imbert B, et al. Real-world evidence for a prescription digital therapeutic to treat opioid use disorder. Curr Med Res Opin. 2021;37(2):175-183. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shi JM, Henry SP, Dwy SL, Orazietti SA, Carroll KM. Randomized pilot trial of web-based cognitive-behavioral therapy adapted for use in office-based buprenorphine maintenance. Subst Abus. 2019;40(2):132-135. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kiluk BD, Nich C, Buck MB, Devore KA, Frankforter TL, LaPaglia DM, et al. Randomized clinical trial of computerized and clinician-delivered CBT in comparison with standard outpatient treatment for substance use disorders: primary within-treatment and follow-up outcomes. Am J Psychiatry. 2018;175(9):853-863. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gustafson DH, Landucci G, McTavish F, Kornfield R, Johnson RA, Mares ML, et al. The effect of bundling medication-assisted treatment for opioid addiction with mHealth: study protocol for a randomized clinical trial. Trials. 2016;17(1):592. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gardiner PM, McCue KD, Negash LM, Cheng T, White LF, Yinusa-Nyahkoon L, et al. Engaging women with an embodied conversational agent to deliver mindfulness and lifestyle recommendations: a feasibility randomized control trial. Patient Educ Couns. 2017;100(9):1720-1729. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gaffney H, Mansell W, Tai S. Conversational agents in the treatment of mental health problems: mixed-method systematic review. JMIR Ment Health. 2019;6(10):e14166. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Boumparis N, Schulte MHJ, Riper H. Digital mental health for alcohol and substance use disorders. Curr Treat Options Psych. 2019;6(4):352-366. [ FREE Full text ] [ CrossRef ]
• Laranjo L, Dunn AG, Tong HL, Kocaballi AB, Chen J, Bashir R, et al. Conversational agents in healthcare: a systematic review. J Am Med Inform Assoc. 2018;25(9):1248-1258. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lucas GM, Gratch J, King A, Morency L. It's only a computer: virtual humans increase willingness to disclose. Comput Hum Behav. 2014;37:94-100. [ CrossRef ]
• Cook JE, Doyle C. Working alliance in online therapy as compared to face-to-face therapy: preliminary results. Cyberpsychol Behav. 2002;5(2):95-105. [ CrossRef ] [ Medline ]
• Berry K, Salter A, Morris R, James S, Bucci S. Assessing therapeutic alliance in the context of mHealth interventions for mental health problems: development of the mobile Agnew Relationship Measure (mARM) questionnaire. J Med Internet Res. 2018;20(4):e90. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Beatty C, Malik T, Meheli S, Sinha C. Evaluating the therapeutic alliance with a free-text CBT conversational agent (Wysa): a mixed-methods study. Front Digit Health. 2022;4:847991. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 17.04.23; peer-reviewed by A Ahmadipour, KL Mauco; comments to author 23.08.23; revised version received 07.12.23; accepted 11.01.24; published 01.04.24.

©Van Lewis King Jr, Gregg Siegel, Henry Richard Priesmeyer, Leslie H Siegel, Jennifer S Potter. Originally published in JMIR Formative Research (https://formative.jmir.org), 01.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.