research paper on web application

Research and Implementation of WEB Application Firewall Based on Feature Matching

• Conference paper
• First Online: 30 March 2019
• pp 1223–1231
• Cite this conference paper

• Hui Yuan 18 ,
• Lei Zheng 18 ,
• Liang Dong 18 ,
• Xiangli Peng 18 ,
• Yan Zhuang 18 &
• Guoru Deng 18  

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 929))

Included in the following conference series:

• International Conference on Application of Intelligent Systems in Multi-modal Information Analytics

1595 Accesses

2 Citations

The rapid development of the Internet has brought great convenience to our lives. Correspondingly, the rapid development of the web relies on the continuous development of Internet infrastructure such as hardware and application software and related protocols. However, with the rapid development of Web applications, the security situation is not optimistic. Most Web applications have security vulnerabilities, and traditional network security devices have limited protection against attacks at the application layer. A traditional firewall can only protect the network layer. IDS and IPS cannot effectively protect against application layer attacks through flexible coding and packet segmentation. Therefore, this paper analyzes HTTP protocol and mainstream Web attacks and their bypass methods. Aiming at the shortcomings of HTTP protocol and pattern matching, this paper proposes a Web application firewall system based on feature matching. Experiments show that the Web application firewall system can defend against various web application layer attacks and effectively solve the omission problem of Web attack detection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Torrano-Gimenez C, Hai TN, Alvarez G et al (2014) Applying feature selection to payload-based Web Application Firewalls. In: Third international workshop on security and communication networks. IEEE, pp 75–81

Google Scholar  

Torrano-Gimenez C, Perez-Villegas A, Álvarez G (2015) An anomaly-based web application firewall. In: Security and cryptography, pp 23–28

Feng F, Shao C, Liu D (2012) Design and implementation of coldfusion-based web application firewall. In: International conference on computer science & service system. IEEE, pp 659–662

Tekerek A, Gemci C, Bay OF (2015) Development of a hybrid web application firewall to prevent web based attacks. In: IEEE international conference on application of information and communication technologies. IEEE, pp 1–4

He WJ, Feng L, Yu J et al (2015) Design of lightweight web application firewall based on cloud push. Inf Secur Technol

Manikanta YVN, Sardana A (2012) Protecting web applications from SQL injection attacks by using framework and database firewall. In: Proceedings of the international conference on advances in computing, communications and informatics. ACM, pp 609–613

Park H (2015) Advanced security configuration options for SAS® 9.4 web applications and mobile devices

Xiang J (2016) Design scheme of campus network based on web application firewall. Shanxi Electron Technol

Wang H, Ma Y, Wang C (2016) The research and improvement of intrusion detection model based on modsecurity firewall. Acta Scientiarum Naturalium Universitatis Nankaiensis

Makiou A, Begriche Y, Serhrouchni A (2015) Improving web application firewalls to detect advanced SQL injection attacks. In: International conference on information assurance and security. IEEE, pp 35–40

Liu Z (2014) Analysis on firewall technologies of web-based application. Inf Res 2014

Banjo I (2016) System and method for event-related content discovery, curation, and presentation

Rostami-Hesarsorkh S, Jacobsen M (2014) Detecting encrypted tunneling traffic: US, US8856910[P]

Fruth RC, Ramdatmisier V, Markey B et al (2017) Passive web application firewall

Gómez C, White JC, Wulder MA et al (2015) Integrated object-based spatiotemporal characterization of forest change from an annual time series of landsat image composites. Can J Remote Sens 41(4):271–292

Article   Google Scholar  

Ghanbari Z, Rahmani Y, Ghaffarian H et al (2016) Comparative approach to web application firewalls. In: International conference on knowledge-based engineering and innovation. IEEE, pp 808–812

Epp N, Funk R, Cappo C (2017) Anomaly-based web application firewall using HTTP-specific features and one-class SVM. In: Workshop Regional de Segurança da Informação e de Sistemas Computacionais

Ji P, Luo L, Sreedhar VC et al (2018) Hierarchical rule development and binding for web application server firewall

Download references

Author information

Authors and affiliations.

Information and Communication Company of Hubei State Grid Corporation, No. 197 Xu Dong Street, Wuhan, Hubei, China

Hui Yuan, Lei Zheng, Liang Dong, Xiangli Peng, Yan Zhuang & Guoru Deng

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Hui Yuan .

Editor information

Editors and affiliations.

Department of Decision and Information Sciences, Oakland University School of Business Administration, Rochester, MI, USA

Vijayan Sugumaran

Shanghai University, Shanghai, China

Vinayaka Mission’s Research Foundation, Aarupadai Veedu Institute of Technology, Salem, Tamil Nadu, India

Department of Informatics, University of Leicester, Leicester, Leicestershire, UK

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Cite this paper.

Yuan, H., Zheng, L., Dong, L., Peng, X., Zhuang, Y., Deng, G. (2019). Research and Implementation of WEB Application Firewall Based on Feature Matching. In: Sugumaran, V., Xu, Z., P., S., Zhou, H. (eds) Application of Intelligent Systems in Multi-modal Information Analytics. MMIA 2019. Advances in Intelligent Systems and Computing, vol 929. Springer, Cham. https://doi.org/10.1007/978-3-030-15740-1_154

Download citation

DOI : https://doi.org/10.1007/978-3-030-15740-1_154

Published : 30 March 2019

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-15738-8

Online ISBN : 978-3-030-15740-1

eBook Packages : Intelligent Technologies and Robotics Intelligent Technologies and Robotics (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

web development Recently Published Documents

Total documents.

• Latest Documents
• Most Cited Documents
• Contributed Authors
• Related Sources
• Related Keywords

Website Developmemt Technologies: A Review

Abstract: Service Science is that the basis of knowledge system and net services that judge to the provider/client model. This paper developments a technique which will be utilized in the event of net services like websites, net applications and eCommerce. The goal is to development a technique that may add structure to a extremely unstructured drawback to help within the development and success of net services. The new methodology projected are going to be referred to as {the net|the online|the net} Development Life Cycle (WDLC) and tailored from existing methodologies and applied to the context of web development. This paper can define well the projected phases of the WDLC. Keywords: Web Development, Application Development, Technologies, eCommerce.

Analysis of Russian Segment of the Web Development Market Operating Online on Upwork

The Russian segment of the web services market in the online environment, on the platform of the Upwork freelance exchange, is considered, its key characteristics, the composition of participants, development trends are highlighted, and the market structure is identified. It is found that despite the low barriers to entry, the web development market is very stable, since the composition of entrenched firms that have been operating for more than six years remains. The pricing policy of most Russian companies indicates that they work in the middle price segment and have low budgets, which is due to the specifics of the foreign market and high competition.

Farming Assistant Web Services: Agricultor

Abstract: Our farming assistant web services provides assistance to new as well as establish farmers to get the solutions to dayto-day problems faced in the field. A farmer gets to connect with other farmers throughout India to get more information about a particular crop which is popular in other states. Keywords: Farmers, Assistance, Web Development

Tradução de ementas e histórico escolar para o inglês: contribuição para participação de discentes do curso técnico em informática para internet integrado ao ensino médio em programas de mobilidade acadêmica / Translation of summary and school records into english: contribution to the participation of high school with associate technical degree on web development students in academic mobility programs

Coded websites vs wordpress websites.

This document gives multiple instructions related to web developers using older as well as newer technology. Websites are being created using newer technologies like wordpress whereas on the other hand many people prefer making websites using the traditional way. This document will clear the doubt whether an individual should use wordpress websites or coded websites according to the users convenience. The Responsiveness of the websites, the use of CMS nowadays, more and more up gradation of technologies with SEO, themes, templates, etc. make things like web development much much easier. The aesthetics, the culture, the expressions, the features all together add up in order make the designing and development a lot more efficient and effective. Digital Marketing has a tremendous growth over the last two years and yet shows no signs of stopping, is closely related with the web development environment. Nowadays all businesses are going online due to which the impact of web development has become such that it has become an integral part of any online business.

Cognitive disabilities and web accessibility: a survey into the Brazilian web development community

Cognitive disabilities include a diversity of conditions related to cognitive functions, such as reading, understanding, learning, solving problems, memorization and speaking. They differ largely from each other, making them a heterogeneous complex set of disabilities. Although the awareness about cognitive disabilities has been increasing in the last few years, it is still less than necessary compared to other disabilities. The need for an investigation about this issue is part of the agenda of the Challenge 2 (Accessibility and Digital Inclusion) from GranDIHC-Br. This paper describes the results of an online exploratory survey conducted with 105 web development professionals from different sectors to understand their knowledge and barriers regarding accessibility for people with cognitive disabilities. The results evidenced three biases that potentially prevent those professionals from approaching cogni-tive disabilities: strong organizational barriers; difficulty to understand user needs related to cognitive disabilities; a knowledge gap about web accessibility principles and guidelines. Our results confirmed that web development professionals are unaware about cognitive disabilities mostly by a lack of knowledge about them, even if they understand web accessibility in a technical level. Therefore, we suggest that applied research studies focus on how to fill this knowledge gap before providing tools, artifacts or frameworks.

PERANCANGAN WEB RESPONSIVE UNTUK SISTEM INFORMASI OBAT-OBATAN

A good information system must not only be neat, effective, and resilient, but also must be user friendly and up to date. In a sense, it is able to be applied to various types of electronic devices, easily accessible at any whereand time (real time), and can be modified according to user needs in a relatively easy and simple way. Information systems are now needed by various parties, especially in the field of administration and sale of medicines for Cut Nyak Dhien Hospital. During this time, recording in books has been very ineffective and caused many problems, such as difficulty in accessing old data, asa well as the information obtained was not real time. To solve it, this research raises the theme of the appropriate information system design for the hospital concerned, by utilizing CSS Bootstrap framework and research methodology for web development, namely Web Development Life Cycle. This research resulted in a responsive system by providing easy access through desktop computers, tablets, and smartphones so that it would help the hospital in the data processing process in real time.

Web Development and performance comparison of Web Development Technologies in Node.js and Python

“tom had us all doing front-end web development”: a nostalgic (re)imagining of myspace, assessment of site classifications according to layout type in web development, export citation format, share document.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

•  We're Hiring!
•  Help Center

Web Applications

• Most Cited Papers
• Most Downloaded Papers
• Newest Papers
• Save to Library
• Last »
• Inventory Management Follow Following
• Happy and Simple Person Follow Following
• Web Development Follow Following
• Web Technologies Follow Following
• Web Design Follow Following
• Computer Science Follow Following
• Web 2.0 Follow Following
• Semantic Web Follow Following
• Australia Follow Following
• Web Engineering Follow Following

Enter the email address you signed up with and we'll email you a reset link.

• Academia.edu Publishing
•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

HTTPS Contribution in Web Application Security: A Systematic Literature Review

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• JMIR Hum Factors
• v.8(1); Jan-Mar 2021

A Web-Based Application for Complex Health Care Populations: User-Centered Design Approach

Francesca ferrucci.

1 Informapro Srl, Rome, Italy

2 Department of Human Sciences, Communication and Tourism, University of Tuscia, Viterbo, Italy

3 EuResist Network European Economic Interest Grouping, Rome, Italy

Manuele Jorio

Stefano marci.

4 Unità Operativa Complessa Materno-Infantile - Azienda Sanitaria Locale Rieti, Consultorio Pediatrico, Rieti, Italy

Antonia Bezenchek

Giulia diella.

5 Academic Department of Pediatrics, Division of Immune and Infectious Diseases - Istituto di Ricovero e Cura a Carattere Scientifico Ospedale Pediatrico Bambino Gesù, Rome, Italy

Cinzia Nulli

Ferdinando miranda, guido castelli-gattinara, associated data.

Screenshots and description of the Italian graphic user interface.

Tested Abilita functions.

Feedback questionnaire—patients.

Feedback questionnaire—health care providers.

Although eHealth technology makes it possible to improve the management of complex health care systems and follow up on chronic patients, it is not without challenges, thus requiring the development of efficient programs and graphic user interface (GUI) features. Similar information technology tools are crucial, as health care populations are going to have to endure social distancing measures in the forthcoming months and years.

This study aims to provide adequate and personalized support to complex health care populations by developing a specific web-based mobile app. The app is designed around the patient and adapted to specific groups, for example, people with complex or rare diseases, autism, or disabilities (especially among children) as well as Alzheimer or senile dementia. The app’s core features include the collection, labeling, analysis, and sorting of clinical data. Furthermore, it authorizes a network of people around the patient to securely access the data contained in his or her electronic health record.

The application was designed according to the paradigms of patient-centered care and user-centered design (UCD). It considers the patient as the main empowered and motivating factor in the management of his or her well-being. Implementation was informed through a family needs and technology perception assessment. We used 3 interdisciplinary focus groups and 2 assessment surveys to study the contexts of app use, subpopulation management, and preferred functions. Finally, we developed an observational study involving 116 enrolled patients and 253 system users, followed by 2 feedback surveys to evaluate the performance and impact of the app.

In the validated general GUI, we developed 10 user profiles with different privacy settings. We tested 81 functions and studied a modular structure based on disease or medical area. This allowed us to identify replicable methods to be applied to module design. The observational study not only showed good family and community engagement but also revealed some limitations that need to be addressed. In total, 42 of 51 (82%) patients described themselves as satisfied or very satisfied . Health care providers reported facilitated communication with colleagues and the need to support data quality.

Conclusions

The experimented solution addressed some of the health system challenges mentioned by the World Health Organization: usability appears to be significantly improved when the GUI is designed according to patients’ UCD mental models and when new media and medical literacy are promoted. This makes it possible to maximize the impact of eHealth products, thereby overcoming some crucial gaps reported in the literature. Two main features seemed to have potential benefit compared with other eHealth products: the modeling, within the app, of both the formal and informal health care support networks and the modular structure allowing for comorbidity management, both of which require further implementation.

Introduction

The improvement in health services and the quality of health treatment and social care has led to a significant increase in survival (and quality of life) among adults and children with chronic complex diseases and high health care needs [ 1 ].

According to the World Health Organization (WHO), over a billion people have some form of disability, whereas 110 to 190 million adults have significant difficulty functioning. An estimated 39% of the Italian population is affected by some chronic disease, with increasing disability rates. Currently, more than 3 million people in Italy are disabled. These patients are characterized by multiple morbidities, requiring the use of a range of services and a technology-enhanced care model [ 1 - 4 ].

eHealth may help such patients manage multiple clinical encounters and large amounts of clinical information generated from various sources. Indeed, patients report a highly frequent use of information and communications technology (ICT) to search for health information, communicate with health care providers (HCPs), track medical information and medications, and assist in decision making regarding treatment [ 5 ]. Notably, patients attempt to use ICT tools for self-management, as they expect to benefit from eHealth and enhance control over their own disease [ 6 ].

Extant research suggests that eHealth tools supporting patient-HCP interaction, patient self-management, and HCP-HCP interactions (through electronic health record integration) are of great benefit to patients [ 7 , 8 ]. These benefits may increase further, as the COVID-19 crisis has triggered additional demand for remote care models and systems. Previous studies have pointed out a number of critical issues concerning complex health care populations, since these include different subpopulations that pose specific medical and organizational challenges for the design of public service provision. These issues include the accurate assessment of the levels of services and needs, implementation of services and resources tailored to specific needs, coordination and integration of family-centered care planning, promotion of health systems based on patient or family self-management, and the redefinition of models of multidisciplinary team care [ 5 , 9 , 10 ].

According to the 2012-2020 eHealth Action Plan, in 2011, the Italian Public Administration promoted a high-communication health care project and a citizen’s Electronic Health Dossier ( Fascicolo Sanitario Elettronico ) [ 8 , 11 ], but the project encountered difficulties in getting under way and proved difficult to implement. The few ongoing initiatives have not received positive feedback from users due to usability problems and the low digital literacy of both HCPs and families [ 12 ].

In this context, the ABILITA2 Project (Italian: Sviluppo di un Applicativo per terminali moBILI dedicato a popolazioni ad alTA complessità Assistenziale ; English: Development of a web-based Mobile Application for complex healthcare populations ) takes advantage of ICT and its eHealth applications, exploiting the patient-centered care approach. When addressing the abovementioned issues, it adapts the service to different subpopulations, providing models that can be replicated in the future [ 13 ].

To meet the requirement of interdisciplinarity, the ABILITA2 consortium includes a partnership between ICT companies (Informapro Srl, Logica Informatica Srl, and Mediamed Interactive Srl) and medical and research centers ( Ospedale Pediatrico Bambino Gesù - Rome and Consultorio Pediatrico ASL Rieti) as well as patient associations related to the medical areas of Alzheimer disease, autism, artificial nutrition, and rare pediatric diseases.

The project’s general objective was to provide adequate and personalized support to complex health care populations by developing a specific web-based app, Abilita , designed around the patient and customizable for specific groups, notably people with complex or rare diseases (eg, genetic syndromes, patients requiring parenteral nutrition), autism or disabilities (especially among children), and Alzheimer or senile dementia. The core features of the app allow for the collection, labeling, analysis, and sorting of clinical data. Furthermore, it authorizes a network of people around the patient to securely access the data contained in his or her electronic health record.

The study’s specific objectives are as follows:

• Assess levels of service and patient needs, testing assessment procedures and tools, especially for pediatric and older adult groups who are less considered in the eHealth market.
• Promote patient self-management and co-responsibility as the basis for a suitable and user-friendly web application. The emphasis is on patient empowerment (understanding of his or her role, acquisition of sufficient knowledge to be able to engage with HCPs, patient skills, and the availability of a facilitating environment [ 14 , 15 ]).
• Enhance and innovate the coordination between professionals and caregivers, specifically exploring the potential of a collaborative network operating on the patient’s behalf, which is built by the patient based on his or her individual needs and institutional contacts.
• Make the most of a proximity support network , which includes informal relationships with relatives, friends, and key figures in the territory, which is a crucial health care management factor [ 16 , 17 ].
• Encourage families or communities to play an active role and, at the same time, ensure quality of data, care, and assistance by using GUI modeling of proper actions per profile according to the level of skill and motivation.
• Assess the app’s performance and impact.

Assessment and Design Process

The project adopted a user-centered design (UCD) approach in graphic user interfaces (GUIs) and considered users’ point of view and needs as central. The difference from other methods is that UCD meets the needs and desires of users rather than forcing them to change their behavior to meet the product settings [ 18 ]. Since the designers considered the user to be the patient (or parent/caregiver), an interdisciplinary analysis was needed to assess needs and then model actions, logic paths, questions, and answers within the interface. To do so, clinical and medical competence needs to be flanked by skills in computer sciences and database management, communication or new media sciences, psychology, and sociology [ 13 ]. The study used a number of focus groups based on a general inductive approach. The results of these focus groups were then further investigated through anonymous questionnaires [ 19 ]. The focus groups met monthly with 90- to 120-min sessions to analyze the different issues raised by the study.

Focus group A assessed patients’ needs and scenarios of use. It included patients (n=4), health care workers (n=2), psychologists (n=1), researchers in communication sciences (n=1), and software developers (n=1). All participants were part of the project network and discussed the experience of patients and caregivers with ICT products and possible scenarios using the Abilita app. Finally, a web-based questionnaire (Q1) was developed for the purpose of studying the main features, habits, needs, and digital and medical literacy of patients and families. Q1 was sent to a selected sample of patient associations (presidents and expert members in steering groups): Alzheimer Uniti Roma ONLUS, Associazione Nazionale Genitori Soggetti Autistici (ANGSA) Lazio Onlus, Associazione italiana sulla nutrizione Artificiale Domiciliare “Un filo per la Vita,” Associazione Prader Willi Lazio, Associazione Italiana delezione cromosoma 22 Onlus . The 20 anonymous responses were collected in June 2018; and the statistics of multiple-choice items and summaries of open-answer items were contained in a project report in September 2018 [ 20 , 21 ].

Focus group B, consisting of HCPs (n=4), psychologists (n=1), privacy officers (n=1), and software developers (n=2), was devoted to the general GUI design. The outcomes of the assessment of patient needs were translated into design challenges. The discussion raised a number of research questions, including the problem of low HCP motivation or time and the need to consider the patient as the main subject motivated to use the app. It is also necessary to task the patient or caregiver with data entry and updating health records and adding user profiles to the app (to model both institutional and informal patient support networks). Additional issues concerned the powers of individual user profiles (reading or writing of sections of the data set), the need to ensure health data quality, even when not directly entered by HCPs, and to predict real-world data entered by the patient and his or her proximity network. We used paper prototyping throughout the process that led to the user requirements document delivered in November 2018 for all identified user profiles (patient, parent or tutor, caregiver, family member, doctor, nurse, structure manager, social operator, temporary, and emergency).

In designing the health record, we tried to identify possible user behaviors, which led to additional questions: what does a particular population require and how can the interface structure be customized for specific pathologies to meet patient needs and coordination requirements? Data and pages are not equally relevant for all subpopulations, and preferred content, information, and functionalities differ across groups. In this respect, the general GUI of Abilita could be made more powerful by customizing content and database structure, with a view to create GUIs for more specific medical areas (the Abilita modules ).

Focus group C was set up to assess this potential. It included presidents and steering group members from patient associations (n=4), psychologists (n=1), communication sciences researchers (n=1), and software developers (n=1). The discussion addressed the specific needs of the subpopulations involved in the study, after which we administered a mandatory questionnaire (Q2) to test the usefulness and effectiveness of feasible implementations. Q2 was sent out through email to a selected sample of national and regional patient associations; the 15 anonymous responses were then collected into a database highlighting the main aspects or attention points for GUI customization and the preferred functions that could be identified.

Observational Study, Feedback, and Validation

After the development of the prototype, we performed an observational study to evaluate its application in terms of its functionality, versatility, responsiveness to patients or families’ needs, user-friendliness, and rate of acceptance. We designed the study in line with international Good Clinical Practice criteria and obtained approval from the ethics committees of the medical centers involved (document protocols 1589_OPBG_2018 and 2474/CE Lazio1).

A total of 116 of the 130 (89.2%) patients invited to participate in the study were included, as they (or their families) possessed the required computer skills. They were recruited in the Rome area and in the Province of Rieti, a setting marked by a variety of health needs and increased geographic isolation due to the 2016 earthquake. During the 6-month study period (January-June 2019), the patients authorized additional user profiles to access their data, namely 32 HCPs, 97 parents, 5 family members, and 3 caregivers, for a total of 253 app users.

We then analyzed individual user accesses to explore the actual use of the app. Frontal, telephone, and web-based tutoring sessions helped the patient participants (or their parents if the patient was aged under 16 years) to complete the registration and browse the app upon uploading their personal data. In June 2019, we developed a voluntary web application feedback questionnaire for patients (Q3) with indicators for evaluating usefulness or satisfaction, privacy, and security impact. We identified usability and effectiveness, while task managers tested the app’s compliance with general recommendations and technical functionality. A link to the questionnaire was sent by email (we avoided multiple responses by limiting survey access to a single instance), and we received 51 anonymous responses in July 2019; the statistics on multiple-choice items and summaries of open-answer items were reported in a project report in September 2019.

In July 2019, we conducted 23 semistructured individual interviews with 10 doctors and 13 nurses to explore the app’s usefulness in the follow-up of chronic patients, its usability, and other features of the HCP interface (questionnaire Q4).

Table 1 summarizes the different data collection stages of the research.

Data collection processes.

a HCP: health care provider.

b GUI: graphic user interface.

Q1 clarified the overall context of the study. The age at first diagnosis for complex health care diseases ranged from 0 to 5 years for the majority of cases and from 65 to 80 years in the remaining cases. All patients were not autonomous and had at least one caregiver. Their digital skills were at a basic or medium level, with limited experience with the use of IT tools to communicate with social and (private or public) health care services. Patients or caregivers displayed significant awareness of their medical areas. They were able to name the diagnosis in technical terms, describe the main elements of the disorder or disease (causes, severity, symptomatology, correlations with other disorders, and risk factors), mention the pharmacological therapies with precision, describe recommended daily treatments and activities (diets, sport), and recognize changes in symptoms (especially aspects to be monitored and reported to health care personnel). The most frequently used documents were treatment plans, reports of visits or exams, and prescriptions. Most patients reported to a health care unit devoted to their specific disorder or disease and scheduled follow-up visits every 6 months on average. In this context, potential clients believed that Abilita could successfully respond to the following requirements:

• Provision of tools and resources to manage emergency situations (average score of 8.2 on a 0-10 scale, SD 1.6).
• Collection and storage of health care documents and digital contents (average score of 7.7 on a 0-10 scale, SD 3.0).
• Remote communication with authorized health care personnel (average score of 7.6 on a 0-10 scale, SD 2.1).
• Support with monitoring activities (reminders of exams, visits, self-measurements, etc; average score of 6.7 on a 0-10 scale, SD 2.8).
• Targeted information on recreational, informative, or social activities (average score of 6.1 on a 0-10 scale, SD 2.3).

Focus group A identified the Online Help function as a central tool for the app, as it served multiple goals: it accompanies the user in browsing the sections even when he or she has low digital or medical literacy, and it acts as an intermediary between the different users operating within a patient’s personal folder.

Focus group B confirmed the main areas of the GUI (menu items) as follows: Home page; Help; My data; My network; Search; My story; Organizer; Notifications; Personal profile; Info room; Emergency card . The Online Help, personalized as a female avatar named Lisa , interacts with the user by written and/or audiovisual messages. The app also features a medical glossary explaining technical terms and jargon. When users first access the app, Lisa provides advice and recommendations on how to start, suggests the sections to be prioritized, and offers easily accessible demos of app functions. In subsequent usage, Lisa highlights unread notifications, scheduled appointments, and missing information in the Emergency card when relevant ( Figure 1 ).

Home page–shortcuts to the main areas and welcome or follow-up message from Lisa.

The my data area is the medical and administrative record and comprises 2 sections: general outline and clinical data and documents ( Figure 2 ). The sections include importance or severity labels that ensure the record’s organization and facilitate access to the most relevant data. Key information on the type of disease, therapy, particular care needs, and specific conditions is easily available. Thanks to the validation function, HCPs can validate data entered by patients or caregivers.

Area “My data.”.

In the area my network , the patient or the parent or legal tutor can create a personalized collaborative network of care support (eg, doctors, nurses, parents, friends, neighbors, domestic helpers, babysitters and tutors, teachers, etc). Each member of the network is assigned a separate profile with authorization to access some or all of the personal data. Furthermore, the patient may authorize all health care facilities, thereby enabling all HCP personnel to read and update their medical records. The app also makes available temporary or emergency authorization facilities as well as the blanket withdrawal of all permissions. In the search area, it is possible to carry out simple or advanced database searches sorted by data subject or by authorized person (highly recommended by HCPs to facilitate access to relevant information). My story hosts a personal diary where users can note clinical data as well as daily experiences, relevant episodes or therapeutic adherence ( Multimedia Appendix 1 ). Actions in the app are always traceable, which allow reconstruction of the author and the date of changes and data validation. Figure 3 summarizes the results of the design process, the relationship between the design and objectives of the research (as discussed in the focus groups) and privacy policy.

The design process.

In keeping with the privacy policy, the patient is the sole owner and controller of his or her data and the only person able to decide who may treat them and under what conditions, which meets both General Data Protection Regulation requirements and recommendations concerning patient empowerment [ 22 , 23 ]. All sensitive data and interactions between the client (web-based application or emergency mobile app) and the server are encrypted.

The results of focus group C confirm that the GUI’s disease specificity crucially improves app usability and patient engagement. The relevance of the data set and the perception of utility by families and communities increases when the app is customized based on the specific needs of a subpopulation. In particular, we studied subpopulation management for the following medical areas: autism spectrum disorders, 22q11.2 deletion syndrome, Alzheimer disease, Prader-Willi syndrome, and chronic intestinal failure. The main gaps were centered around the coordination of social and health care services (mostly during follow-up) as well as family support. As a result, the design of the Abilita modules for each medical area includes specific GUI features: personalization of the content and structure of the medical data set, contents of the info room (information about the disease), and functions of the organizer and notifications as well as recommendations and priority highlights from Lisa. More specifically, the study foregrounded the following elements:

• Each subpopulation would like to have a personalized page in the clinical data subsection.
• Different diseases and ages need differentiated administrative forms.
• The agenda and remind functions could be implemented for specific situations and connected with local networks.
• Users consider it important that data for clinical research at different levels be available.
• Users consider the latest disease-specific documents and recommendations important, such as the Integrated Care Pathway or best clinical practices.

Table 2 shows the characteristics of enrolled patients and families as well as their average use of the Abilita app over the last 4 to 6 months of study. These data were automatically exported by the system administrators and reflect the actions performed by users within the app, including demographic data entered at registration.

Statistics of use of the study population (N=116).

Owing to the characteristics of the investigators (pediatricians), most of the enrolled subjects were children or adolescents, in which case the users of the app were mainly parents or family members. HCPs authorized by patients or parents primarily uploaded clinical data and documents. Patients performed operations such as consultation with clinical data, loading of missing clinical investigations, and writing of individual day-to-day experiences. Each patient authorized an average of approximately 2 persons to access their data, who were usually parents and family members, doctors, nurses, and psychologists. By contrast, caregivers and school operators were considerably less involved. The 868 documents that were uploaded included 18 different subtypes, mainly reports of examinations and clinical investigations. Approximately 35% of the data entries were performed by the patients or their parents from the beginning.

We tested 81 Abilita functions, which users could access with different levels of authorization ( Multimedia Appendix 2 ). Q3 involved 51 respondents. Table 3 shows the results of the answers to questions 1 to 16, with average positive scores of 78% (4 or 5).

Answers to questions 1-16, expressed in percentage of Likert scale scores.

Questions 17 and 18 asked users about the areas they would like to see enhanced: the answers covered all the areas suggested, with no specific option prevailing significantly, and the same applies to what functions should be integrated (question 18). Interestingly, the option ability to set preferred tabs or activities to create shortcuts for most used functions obtained 37% (19/51) of the responses, suggesting that customization is the best strategy. No relevant issues arose regarding privacy and security (questions 19-20): 57% (29/51) of users had no general problems, 65% (33/51) had no problems entering and classifying data, only 23% (12/51) had problems but overcame them with the Lisa online help or with practice (questions 21-30).

Other open and unstructured optional questions (31-36) yielded good feedback concerning the Lisa web-based help, with 47% (24/51) suggesting further implementation of this tool. Patients and caregivers urged informing family doctors and pediatricians about the app to maximize dissemination. The answers on scientific research and on PDTAs (diagnostic-therapeutic assistance pathways) highlight Abilita ’s potential for data collection subject to privacy consent, for reconstructing analogies in groups of patients affected by the same disease or disorder, and for patient associations to pursue their institutional goals. In addition, Abilita ’s effectiveness in facilitating relationships or communication with HCPs and local facilities was positively evaluated, preferably with the support of the region. Furthermore, participants considered that the main strengths of the project were knowledge of one's own medical history with a click and the overall philosophy behind the app ( Multimedia Appendix 3 ).

Q4, which included 17 predefined questions and addressed 23 HCPs, produced average positive scores of 72% (4 or 5) in the first 16 items defined by a Likert scale score ( Multimedia Appendix 4 ). In the last open-answer item, asking strengths or weaknesses of the project, the following aspects were highlighted:

• The availability of reports and alerts facilitated communication among HCPs and accelerated diagnostic and care paths.
• Users appreciated the involvement of patients or parents in the data entry of documents, lab results, and parameters, although 6 respondents raised concern about quality.
• Overall, 39% (9/23) of respondents encountered general problems in using Abilita , especially in the first weeks, and asked that Online Help tools be implemented.
• Users appreciated the importance or severity labels.

Principal Findings

The project used needs assessment to establish the contexts to interface with, showing a prevalence of non–self-sufficient patients—typically infants and older adults—diagnosed at an average age of 0 to 5 or 65 to 80 years and mainly supported by health care units specifically devoted to the disorder or disease, for whom follow-up visits are scheduled on average every 6 months. Basic digital skills and good levels of medical literacy of families were identified as starting points of the design.

A sample of 116 patients participated in the observational study. Each patient authorized an average of 1.8 persons to access his or her data, typically parents and family members, doctors, nurses, and psychologists, with the additional involvement of the communities of other institutions and informal environments, for a total of 253 system users. In approximately 35% of cases, data entry was performed by the patients or their parents from the beginning.

Questionnaire Q3 yielded positive patient feedback on the utility of the app to address some health system challenges mentioned as relevant by WHO [ 24 ] and on themes such as delayed reporting of events (WHO challenge 1.2), communication roadblocks, lack of access to information or data, insufficient utilization of data and information (WHO challenges 1.4-1.6), insufficient continuity of care, inadequate supportive supervision (WHO challenges 3.5-3.6), low adherence to treatments, and loss of follow-up (WHO challenges 5.2-5.4).

We received no direct evidence on other challenges mentioned by WHO, such as low health worker motivation (3.4), geographic inaccessibility (5.2), insufficient patient engagement (8.1), or absence of community feedback mechanisms (8.3). Some useful indications do emerge in the interpretation of the answers to the same questionnaire Q3. The app promoted communication and team management among HCPs, health care bodies, and families (question 34) and, in addition, increased end user confidence in their own capacity to provide up-to-date, readily searchable, and clear medical information (question 36). According to answers to questions 33 and 35, Abilita can contribute to scientific research and PDTA definition (diagnostic-therapeutic assistance pathways), thereby addressing the lack of population denominator (challenge 1.1) —that is, once used by a larger sample of patients in the same medical area, it can become a tool for further assessment of subpopulation management.

The general choices of the GUI design revealed some advantages:

• The GUI is designed around the patient, who is modeled as the main empowered and motivating actor of the actions necessary to maintain and update the medical record.
• Users are constantly supported by the Online Help (avatar Lisa ), thus addressing medical and digital literacy issues and patient’s commitment in terms of his or her specific role, the main problems that arise while using many ICT products.
• Coordination and management needs can be modeled as pathways and actions recommended by Lisa within the app; they are also addressed by targeted functions (search, calendar, and notification areas).
• Real-world data can be traced and collected to then be reused to advance research on the management of complex chronic conditions.

The issue of data quality, indeed highlighted by 6 of the respondents to the HCP survey, was addressed in the project through the track changes and validation functions. It is worth noting that patients and families are increasingly being required to participate in health monitoring, through daily self-measurement and recording of symptoms or in questionnaires, for diseases such as diabetes, and most recently in the COVID-19 pandemic [ 25 , 26 ]. eHealth market engagement strategies—especially in light of the new patient co-responsibility paradigm—are based on flexibility and customization, with a user-friendly design that makes it possible to communicate with or forward information or data to HCPs [ 27 ]. In its adoption of these strategies, Abilita is in line with a reframed relationship between active citizens and professionals and is intended as a social innovator in the development of a smart community model with the involvement of the proximity network–the app’s core feature.

Although informal or territorial networks were not fully exploited by the users during the observational study, as suggested by the number of authorized user profiles ( Table 2 ), we can hypothesize that this was influenced by the study’s short duration and the characteristics of the patients involved, mainly children and teenagers. The lockdown period in Italy and Europe revealed the need to innovate public health systems precisely in this direction, linking them to local support networks (through new professional figures such as community nurses) and moving toward an integrated vision of health care. The role of volunteering and associations in providing support to self-isolated and vulnerable persons has also been highlighted [ 28 , 29 ]. In this context, specific design choices may require further refinement, considering, for example, the addition of other user profiles such as territory medicine physician or volunteer.

The modular structure of Abilita allows for the personalization of data sets and functions. It also facilitates far-sighted and sustainable investments owing to the partnership’s commercial initiatives, which are aimed at developing new modules (optimal feedback has already been received from relevant stakeholders) and intercepting specific target audiences interested in them. Most importantly, this structure allows the patient to choose one or more application modules in the case of different pathologies. In this way, Abilita has the added value of comorbidity management that is crucial to complex health care populations.

Usability appears to be significantly improved when the GUI is designed according to patients’ mental models and when new media and medical literacy are promoted. Following this principle, the assessment of specific subpopulation needs and the development of personalized GUIs for specific medical areas appears important. Procedures to assess patients’ needs were successfully experimented and a replicable methodology was defined.

Limitations

This analysis was limited by the low number of enrolled subjects and its short duration. Data collected during the study period and answers to questionnaire Q3 refer mainly to pediatric populations; more evidence is needed about older adult patients’ feedback. In fact, only one quarter of them were adults or seniors, but the app was designed and particularly valid for non–self-sufficient subjects, both children and older adults.

The strategy of modular implementation appears to be the best one, but no module has yet been developed and tested. A complete comparison with other available apps, mainly focused on a single disease, will be relevant once the corresponding modules are developed. Specific GUI design choices need to be refined. Nevertheless, the study shows the versatility of this approach for complex health care populations.

eHealth technology allows better management of complex health care aspects in the follow-up of chronic complex disease patients, but translating the UCD into GUI features of an eHealth app is a difficult task. The decision to use patient self-management and co-responsibility as the basis for an eHealth information system seems to have been successful in enhancing the probability of matching the needs of the target population. Moreover, usability appears to be significantly improved when the GUI is designed according to patients’ UCD mental models and when new media and medical literacy are promoted. Its potential applications in an era of greater sociosanitary distancing are certainly of particular interest.

Possible lines of exploitation are as follows:

• Design and develop new Abilita modules dedicated to specific clinical areas with particular care needs (not least with automatic data download and information managed by the patient’s clinical facility of reference).
• Make Abilita an integral part of the automatic distribution of data and dissemination of procedures in the public sector (The Italian National Health Care system is structured by regional area, with disease-specific health care facilities that may be very distant from users).
• Strengthen and expand Abilita and the patient association network to share information and solutions to the various problems faced by caregivers on a daily basis.
• Simplify usability as much as possible with the possible introduction of voice command shortcuts.

Acknowledgments

The authors thank all the patients and families who agreed to participate in this study. They also thank Dr Alberto E Tozzi for his useful and timely suggestions.

Abbreviations

Multimedia appendix 1, multimedia appendix 2, multimedia appendix 3, multimedia appendix 4.

Conflicts of Interest: None declared.

This paper is in the following e-collection/theme issue:

Published on 12.4.2024 in Vol 26 (2024)

Application of AI in in Multilevel Pain Assessment Using Facial Images: Systematic Review and Meta-Analysis

Authors of this article:

• Jian Huo 1 * , MSc   ; 
• Yan Yu 2 * , MMS   ; 
• Wei Lin 3 , MMS   ; 
• Anmin Hu 2, 3, 4 , MMS   ; 
• Chaoran Wu 2 , MD, PhD  

1 Boston Intelligent Medical Research Center, Shenzhen United Scheme Technology Company Limited, Boston, MA, United States

2 Department of Anesthesia, Shenzhen People's Hospital, The First Affiliated Hospital of Southern University of Science and Technology, Shenzhen Key Medical Discipline, Shenzhen, China

3 Shenzhen United Scheme Technology Company Limited, Shenzhen, China

4 The Second Clinical Medical College, Jinan University, Shenzhen, China

*these authors contributed equally

Corresponding Author:

Chaoran Wu, MD, PhD

Department of Anesthesia

Shenzhen People's Hospital, The First Affiliated Hospital of Southern University of Science and Technology

Shenzhen Key Medical Discipline

No 1017, Dongmen North Road

Shenzhen, 518020

Phone: 86 18100282848

Email: [email protected]

Background: The continuous monitoring and recording of patients’ pain status is a major problem in current research on postoperative pain management. In the large number of original or review articles focusing on different approaches for pain assessment, many researchers have investigated how computer vision (CV) can help by capturing facial expressions. However, there is a lack of proper comparison of results between studies to identify current research gaps.

Objective: The purpose of this systematic review and meta-analysis was to investigate the diagnostic performance of artificial intelligence models for multilevel pain assessment from facial images.

Methods: The PubMed, Embase, IEEE, Web of Science, and Cochrane Library databases were searched for related publications before September 30, 2023. Studies that used facial images alone to estimate multiple pain values were included in the systematic review. A study quality assessment was conducted using the Quality Assessment of Diagnostic Accuracy Studies, 2nd edition tool. The performance of these studies was assessed by metrics including sensitivity, specificity, log diagnostic odds ratio (LDOR), and area under the curve (AUC). The intermodal variability was assessed and presented by forest plots.

Results: A total of 45 reports were included in the systematic review. The reported test accuracies ranged from 0.27-0.99, and the other metrics, including the mean standard error (MSE), mean absolute error (MAE), intraclass correlation coefficient (ICC), and Pearson correlation coefficient (PCC), ranged from 0.31-4.61, 0.24-2.8, 0.19-0.83, and 0.48-0.92, respectively. In total, 6 studies were included in the meta-analysis. Their combined sensitivity was 98% (95% CI 96%-99%), specificity was 98% (95% CI 97%-99%), LDOR was 7.99 (95% CI 6.73-9.31), and AUC was 0.99 (95% CI 0.99-1). The subgroup analysis showed that the diagnostic performance was acceptable, although imbalanced data were still emphasized as a major problem. All studies had at least one domain with a high risk of bias, and for 20% (9/45) of studies, there were no applicability concerns.

Conclusions: This review summarizes recent evidence in automatic multilevel pain estimation from facial expressions and compared the test accuracy of results in a meta-analysis. Promising performance for pain estimation from facial images was established by current CV algorithms. Weaknesses in current studies were also identified, suggesting that larger databases and metrics evaluating multiclass classification performance could improve future studies.

Trial Registration: PROSPERO CRD42023418181; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=418181

Introduction

The definition of pain was revised to “an unpleasant sensory and emotional experience associated with, or resembling that associated with, actual or potential tissue damage” in 2020 [ 1 ]. Acute postoperative pain management is important, as pain intensity and duration are critical influencing factors for the transition of acute pain to chronic postsurgical pain [ 2 ]. To avoid the development of chronic pain, guidelines were promoted and discussed to ensure safe and adequate pain relief for patients, and clinicians were recommended to use a validated pain assessment tool to track patients’ responses [ 3 ]. However, these tools, to some extent, depend on communication between physicians and patients, and continuous data cannot be provided [ 4 ]. The continuous assessment and recording of patient pain intensity will not only reduce caregiver burden but also provide data for chronic pain research. Therefore, automatic and accurate pain measurements are necessary.

Researchers have proposed different approaches to measuring pain intensity. Physiological signals, for example, electroencephalography and electromyography, have been used to estimate pain [ 5 - 7 ]. However, it was reported that current pain assessment from physiological signals has difficulties isolating stress and pain with machine learning techniques, as they share conceptual and physiological similarities [ 8 ]. Recent studies have also investigated pain assessment tools for certain patient subgroups. For example, people with deafness or an intellectual disability may not be able to communicate well with nurses, and an objective pain evaluation would be a better option [ 9 , 10 ]. Measuring pain intensity from patient behaviors, such as facial expressions, is also promising for most patients [ 4 ]. As the most comfortable and convenient method, computer vision techniques require no attachments to patients and can monitor multiple participants using 1 device [ 4 ]. However, pain intensity, which is important for pain research, is often not reported.

With the growing trend of assessing pain intensity using artificial intelligence (AI), it is necessary to summarize current publications to determine the strengths and gaps of current studies. Existing research has reviewed machine learning applications for acute postoperative pain prediction, continuous pain detection, and pain intensity estimation [ 10 - 14 ]. Input modalities, including facial recordings and physiological signals such as electroencephalography and electromyography, were also reviewed [ 5 , 8 ]. There have also been studies focusing on deep learning approaches [ 11 ]. AI was applied in children and infant pain evaluation as well [ 15 , 16 ]. However, no study has focused on pain intensity measurement, and no comparison of test accuracy results has been made.

Current AI applications in pain research can be categorized into 3 types: pain assessment, pain prediction and decision support, and pain self-management [ 14 ]. We consider accurate and automatic pain assessment to be the most important area and the foundation of future pain research. In this study, we performed a systematic review and meta-analysis to assess the diagnostic performance of current publications for multilevel pain evaluation.

This study was registered with PROSPERO (International Prospective Register of Systematic Reviews; CRD42023418181) and carried out strictly following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [ 17 ] .

Study Eligibility

Studies that reported AI techniques for multiclass pain intensity classification were eligible. Records including nonhuman or infant participants or 2-class pain detection were excluded. Only studies using facial images of the test participants were accepted. Clinically used pain assessment tools, such as the visual analog scale (VAS) and numerical rating scale (NRS), and other pain intensity indicators, were rejected in the meta-analysis. Textbox 1 presents the eligibility criteria.

Study characteristics and inclusion criteria

• Participants: children and adults aged 12 months or older
• Setting: no restrictions
• Index test: artificial intelligence models that measure pain intensity from facial images
• Reference standard: no restrictions for systematic review; Prkachin and Solomon pain intensity score for meta-analysis
• Study design: no need to specify

Study characteristics and exclusion criteria

• Participants: infants aged 12 months or younger and animal subjects
• Setting: no need to specify
• Index test: studies that use other information such as physiological signals
• Reference standard: other pain evaluation tools, e.g., NRS, VAS, were excluded from meta-analysis
• Study design: reviews

Report characteristics and inclusion criteria

• Year: published between January 1, 2012, and September 30, 2023
• Language: English only
• Publication status: published
• Test accuracy metrics: no restrictions for systematic reviews; studies that reported contingency tables were included for meta-analysis

Report characteristics and exclusion criteria

• Year: no need to specify
• Language: no need to specify
• Publication status: preprints not accepted
• Test accuracy metrics: studies that reported insufficient metrics were excluded from meta-analysis

Search Strategy

In this systematic review, databases including PubMed, Embase, IEEE, Web of Science, and the Cochrane Library were searched until December 2022, and no restrictions were applied. Keywords were “artificial intelligence” AND “pain recognition.” Multimedia Appendix 1 shows the detailed search strategy.

Data Extraction

A total of 2 viewers screened titles and abstracts and selected eligible records independently to assess eligibility, and disagreements were solved by discussion with a third collaborator. A consentient data extraction sheet was prespecified and used to summarize study characteristics independently. Table S5 in Multimedia Appendix 1 shows the detailed items and explanations for data extraction. Diagnostic accuracy data were extracted into contingency tables, including true positives, false positives, false negatives, and true negatives. The data were used to calculate the pooled diagnostic performance of the different models. Some studies included multiple models, and these models were considered independent of each other.

Study Quality Assessment

All included studies were independently assessed by 2 viewers using the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) tool [ 18 ]. QUADAS-2 assesses bias risk across 4 domains, which are patient selection, index test, reference standard, and flow and timing. The first 3 domains are also assessed for applicability concerns. In the systematic review, a specific extension of QUADAS-2, namely, QUADAS-AI, was used to specify the signaling questions [ 19 ].

Meta-Analysis

Meta-analyses were conducted between different AI models. Models with different algorithms or training data were considered different. To evaluate the performance differences between models, the contingency tables during model validation were extracted. Studies that did not report enough diagnostic accuracy data were excluded from meta-analysis.

Hierarchical summary receiver operating characteristic (SROC) curves were fitted to evaluate the diagnostic performance of AI models. These curves were plotted with 95% CIs and prediction regions around averaged sensitivity, specificity, and area under the curve estimates. Heterogeneity was assessed visually by forest plots. A funnel plot was constructed to evaluate the risk of bias.

Subgroup meta-analyses were conducted to evaluate the performance differences at both the model level and task level, and subgroups were created based on different tasks and the proportion of positive and negative samples.

All statistical analyses and plots were produced using RStudio (version 4.2.2; R Core Team) and the R package meta4diag (version 2.1.1; Guo J and Riebler A) [ 20 ].

Study Selection and Included Study Characteristics

A flow diagram representing the study selection process is shown in ( Figure 1 ). After removing 1039 duplicates, the titles and abstracts of a total of 5653 papers were screened, and the percentage agreement of title or abstract screening was 97%. After screening, 51 full-text reports were assessed for eligibility, among which 45 reports were included in the systematic review [ 21 - 65 ]. The percentage agreement of the full-text review was 87%. In 40 of the included studies, contingency tables could not be made. Meta-analyses were conducted based on 8 AI models extracted from 6 studies. Individual study characteristics included in the systematic review are provided in Tables 1 and 2 . The facial feature extraction method can be categorized into 2 classes: geometrical features (GFs) and deep features (DFs). One typical method of extracting GFs is to calculate the distance between facial landmarks. DFs are usually extracted by convolution operations. A total of 20 studies included temporal information, but most of them (18) extracted temporal information through the 3D convolution of video sequences. Feature transformation was also commonly applied to reduce the time for training or fuse features extracted by different methods before inputting them into the classifier. For classifiers, support vector machines (SVMs) and convolutional neural networks (CNNs) were mostly used. Table 1 presents the model designs of the included studies.

a No temporal features are shown by – symbol, time information extracted from 2 images at different time by +, and deep temporal features extracted through the convolution of video sequences by ++.

b SVM: support vector machine.

c GF: geometric feature.

d GMM: gaussian mixture model.

e TPS: thin plate spline.

f DML: distance metric learning.

g MDML: multiview distance metric learning.

h AAM: active appearance model.

i RVR: relevance vector regressor.

j PSPI: Prkachin and Solomon pain intensity.

k I-FES: individual facial expressiveness score.

l LSTM: long short-term memory.

m HCRF: hidden conditional random field.

n GLMM: generalized linear mixed model.

o VLAD: vector of locally aggregated descriptor.

p SVR: support vector regression.

q MDS: multidimensional scaling.

r ELM: extreme learning machine.

s Labeled to distinguish different architectures of ensembled deep learning models.

t DCNN: deep convolutional neural network.

u GSM: gaussian scale mixture.

v DOML: distance ordering metric learning.

w LIAN: locality and identity aware network.

x BiLSTM: bidirectional long short-term memory.

a UNBC: University of Northern British Columbia-McMaster shoulder pain expression archive database.

b LOSO: leave one subject out cross-validation.

c ICC: intraclass correlation coefficient.

d CT: contingency table.

e AUC: area under the curve.

f MSE: mean standard error.

g PCC: Pearson correlation coefficient.

h RMSE: root mean standard error.

i MAE: mean absolute error.

j ICC: intraclass coefficient.

k CCC: concordance correlation coefficient.

l Reported both external and internal validation results and summarized as intervals.

Table 2 summarizes the characteristics of model training and validation. Most studies used publicly available databases, for example, the University of Northern British Columbia-McMaster shoulder pain expression archive database [ 57 ]. Table S4 in Multimedia Appendix 1 summarizes the public databases. A total of 7 studies used self-prepared databases. Frames from video sequences were the most used test objects, as 37 studies output frame-level pain intensity, while few measure pain intensity from video sequences or photos. It was common that a study redefined pain levels to have fewer classes than ground-truth labels. For model validation, cross-validation and leave-one-subject-out validation were commonly used. Only 3 studies performed external validation. For reporting test accuracies, different evaluation metrics were used, including sensitivity, specificity, mean absolute error (MAE), mean standard error (MSE), Pearson correlation coefficient (PCC), and intraclass coefficient (ICC).

Methodological Quality of Included Studies

Table S2 in Multimedia Appendix 1 presents the study quality summary, as assessed by QUADAS-2. There was a risk of bias in all studies, specifically in terms of patient selection, caused by 2 issues. First, the training data are highly imbalanced, and any method to adjust the data distribution may introduce bias. Next, the QUADAS-AI correspondence letter [ 19 ] specifies that preprocessing of images that changes the image size or resolution may introduce bias. However, the applicability concern is low, as the images properly represent the feeling of pain. Studies that used cross-fold validation or leave-one-out cross-validation were considered to have a low risk of bias. Although the Prkachin and Solomon pain intensity (PSPI) score was used by most of the studies, its ability to represent individual pain levels was not clinically validated; as such, the risk of bias and applicability concerns were considered high when the PSPI score was used as the index test. As an advantage of computer vision techniques, the time interval between the index tests was short and was assessed as having a low risk of bias. Risk proportions are shown in Figure 2 . For all 315 entries, 39% (124) were assessed as high-risk. In total, 5 studies had the lowest risk of bias, with 6 domains assessed as low risk [ 26 , 27 , 31 , 32 , 59 ].

Pooled Performance of Included Models

In 6 studies included in the meta-analysis, there were 8 different models. The characteristics of these models are summarized in Table S1 in Multimedia Appendix 2 [ 23 , 24 , 26 , 32 , 41 , 57 ]. Classification of PSPI scores greater than 0, 2, 3, 6, and 9 was selected and considered as different tasks to create contingency tables. The test performance is shown in Figure 3 as hierarchical SROC curves; 27 contingency tables were extracted from 8 models. The sensitivity, specificity, and LDOR were calculated, and the combined sensitivity was 98% (95% CI 96%-99%), the specificity was 98% (95% CI 97%-99%), the LDOR was 7.99 (95% CI 6.73-9.31) and the AUC was 0.99 (95% CI 0.99-1).

Subgroup Analysis

In this study, subgroup analysis was conducted to investigate the performance differences within models. A total of 8 models were separated and summarized as a forest plot in Multimedia Appendix 3 [ 23 , 24 , 26 , 32 , 41 , 57 ]. For model 1, the pooled sensitivity, specificity, and LDOR were 95% (95% CI 86%-99%), 99% (95% CI 98%-100%), and 8.38 (95% CI 6.09-11.19), respectively. For model 2, the pooled sensitivity, specificity, and LDOR were 94% (95% CI 84%-99%), 95% (95% CI 88%-99%), and 6.23 (95% CI 3.52-9.04), respectively. For model 3, the pooled sensitivity, specificity, and LDOR were 100% (95% CI 99%-100%), 100% (95% CI 99%-100%), and 11.55% (95% CI 8.82-14.43), respectively. For model 4, the pooled sensitivity, specificity, and LDOR were 83% (95% CI 43%-99%), 94% (95% CI 79%-99%), and 5.14 (95% CI 0.93-9.31), respectively. For model 5, the pooled sensitivity, specificity, and LDOR were 92% (95% CI 68%-99%), 94% (95% CI 78%-99%), and 6.12 (95% CI 1.82-10.16), respectively. For model 6, the pooled sensitivity, specificity, and LDOR were 94% (95% CI 74%-100%), 94% (95% CI 78%-99%), and 6.59 (95% CI 2.21-11.13), respectively. For model 7, the pooled sensitivity, specificity, and LDOR were 98% (95% CI 90%-100%), 97% (95% CI 87%-100%), and 8.31 (95% CI 4.3-12.29), respectively. For model 8, the pooled sensitivity, specificity, and LDOR were 98% (95% CI 93%-100%), 97% (95% CI 88%-100%), and 8.65 (95% CI 4.84-12.67), respectively.

Heterogeneity Analysis

The meta-analysis results indicated that AI models are applicable for estimating pain intensity from facial images. However, extreme heterogeneity existed within the models except for models 3 and 5, which were proposed by Rathee and Ganotra [ 24 ] and Semwal and Londhe [ 32 ]. A funnel plot is presented in Figure 4 . A high risk of bias was observed.

Pain management has long been a critical problem in clinical practice, and the use of AI may be a solution. For acute pain management, automatic measurement of pain can reduce the burden on caregivers and provide timely warnings. For chronic pain management, as specified by Glare et al [ 2 ], further research is needed, and measurements of pain presence, intensity, and quality are one of the issues to be solved for chronic pain studies. Computer vision could improve pain monitoring through real-time detection for clinical use and data recording for prospective pain studies. To our knowledge, this is the first meta-analysis dedicated to AI performance in multilevel pain level classification.

In this study, one model’s performance at specific pain levels was described by stacking multiple classes into one to make each task a binary classification problem. After careful selection in both the medical and engineering databases, we observed promising results of AI in evaluating multilevel pain intensity through facial images, with high sensitivity (98%), specificity (98%), LDOR (7.99), and AUC (0.99). It is reasonable to believe that AI can accurately evaluate pain intensity from facial images. Moreover, the study quality and risk of bias were evaluated using an adapted QUADAS-2 assessment tool, which is a strength of this study.

To investigate the source of heterogeneity, it was assumed that a well-designed model should have familiar size effects regarding different levels, and a subgroup meta-analysis was conducted. The funnel and forest plots exhibited extreme heterogeneity. The model’s performance at specific pain levels was described and summarized by a forest plot. Within-model heterogeneity was observed in Multimedia Appendix 3 [ 23 , 24 , 26 , 32 , 41 , 57 ] except for 2 models. Models 3 and 5 were different in many aspects, including their algorithms and validation methods, but were both trained with a relatively small data set, and the proportion of positive and negative classes was relatively close to 1. Because training with imbalanced data is a critical problem in computer vision studies [ 66 ], for example, in the University of Northern British Columbia-McMaster pain data set, fewer than 10 frames out of 48,398 had a PSPI score greater than 13. Here, we emphasized that imbalanced data sets are one major cause of heterogeneity, resulting in the poorer performance of AI algorithms.

We tentatively propose a method to minimize the effect of training with imbalanced data by stacking multiple classes into one class, which is already presented in studies included in the systematic review [ 26 , 32 , 42 , 57 ]. Common methods to minimize bias include resampling and data augmentation [ 66 ]. This proposed method is used in the meta-analysis to compare the test results of different studies as well. The stacking method is available when classes are only different in intensity. A disadvantage of combined classes is that the model would be insufficient in clinical practice when the number of classes is low. Commonly used pain evaluation tools, such as VAS, have 10 discrete levels. It is recommended that future studies set the number of pain levels to be at least 10 for model training.

This study is limited for several reasons. First, insufficient data were included because different performance metrics (mean standard error and mean average error) were used in most studies, which could not be summarized into a contingency table. To create a contingency table that can be included in a meta-analysis, the study should report the following: the number of objects used in each pain class for model validation, and the accuracy, sensitivity, specificity, and F 1 -score for each pain class. This table cannot be created if a study reports the MAE, PCC, and other commonly used metrics in AI development. Second, a small study effect was observed in the funnel plot, and the heterogeneity could not be minimized. Another limitation is that the PSPI score is not clinically validated and is not the only tool that assesses pain from facial expressions. There are other clinically validated pain intensity assessment methods, such as the Faces Pain Scale-revised, Wong-Baker Faces Pain Rating Scale, and Oucher Scale [ 3 ]. More databases could be created based on the above-mentioned tools. Finally, AI-assisted pain assessments were supposed to cover larger populations, including incommunicable patients, for example, patients with dementia or patients with masked faces. However, only 1 study considered patients with dementia, which was also caused by limited databases [ 50 ].

AI is a promising tool that can help in pain research in the future. In this systematic review and meta-analysis, one approach using computer vision was investigated to measure pain intensity from facial images. Despite some risk of bias and applicability concerns, CV models can achieve excellent test accuracy. Finally, more CV studies in pain estimation, reporting accuracy in contingency tables, and more pain databases are encouraged for future studies. Specifically, the creation of a balanced public database that contains not only healthy but also nonhealthy participants should be prioritized. The recording process would be better in a clinical environment. Then, it is recommended that researchers report the validation results in terms of accuracy, sensitivity, specificity, or contingency tables, as well as the number of objects for each pain class, for the inclusion of a meta-analysis.

Acknowledgments

WL, AH, and CW contributed to the literature search and data extraction. JH and YY wrote the first draft of the manuscript. All authors contributed to the conception and design of the study, the risk of bias evaluation, data analysis and interpretation, and contributed to and approved the final version of the manuscript.

Data Availability

The data sets generated during and analyzed during this study are available in the Figshare repository [ 67 ].

Conflicts of Interest

None declared.

PRISMA checklist, risk of bias summary, search strategy, database summary and reported items and explanations.

Study performance summary.

Forest plot presenting pooled performance of subgroups in meta-analysis.

• Raja SN, Carr DB, Cohen M, Finnerup NB, Flor H, Gibson S, et al. The revised International Association for the Study of Pain definition of pain: concepts, challenges, and compromises. Pain. 2020;161(9):1976-1982. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Glare P, Aubrey KR, Myles PS. Transition from acute to chronic pain after surgery. Lancet. 2019;393(10180):1537-1546. [ CrossRef ] [ Medline ]
• Chou R, Gordon DB, de Leon-Casasola OA, Rosenberg JM, Bickler S, Brennan T, et al. Management of postoperative pain: a clinical practice guideline from the American Pain Society, the American Society of Regional Anesthesia and Pain Medicine, and the American Society of Anesthesiologists' Committee on Regional Anesthesia, Executive Committee, and Administrative Council. J Pain. 2016;17(2):131-157. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hassan T, Seus D, Wollenberg J, Weitz K, Kunz M, Lautenbacher S, et al. Automatic detection of pain from facial expressions: a survey. IEEE Trans Pattern Anal Mach Intell. 2021;43(6):1815-1831. [ CrossRef ] [ Medline ]
• Mussigmann T, Bardel B, Lefaucheur JP. Resting-State Electroencephalography (EEG) biomarkers of chronic neuropathic pain. A systematic review. Neuroimage. 2022;258:119351. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moscato S, Cortelli P, Chiari L. Physiological responses to pain in cancer patients: a systematic review. Comput Methods Programs Biomed. 2022;217:106682. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Thiam P, Hihn H, Braun DA, Kestler HA, Schwenker F. Multi-modal pain intensity assessment based on physiological signals: a deep learning perspective. Front Physiol. 2021;12:720464. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rojas RF, Brown N, Waddington G, Goecke R. A systematic review of neurophysiological sensing for the assessment of acute pain. NPJ Digit Med. 2023;6(1):76. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mansutti I, Tomé-Pires C, Chiappinotto S, Palese A. Facilitating pain assessment and communication in people with deafness: a systematic review. BMC Public Health. 2023;23(1):1594. [ FREE Full text ] [ CrossRef ] [ Medline ]
• El-Tallawy SN, Ahmed RS, Nagiub MS. Pain management in the most vulnerable intellectual disability: a review. Pain Ther. 2023;12(4):939-961. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gkikas S, Tsiknakis M. Automatic assessment of pain based on deep learning methods: a systematic review. Comput Methods Programs Biomed. 2023;231:107365. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Borna S, Haider CR, Maita KC, Torres RA, Avila FR, Garcia JP, et al. A review of voice-based pain detection in adults using artificial intelligence. Bioengineering (Basel). 2023;10(4):500. [ FREE Full text ] [ CrossRef ] [ Medline ]
• De Sario GD, Haider CR, Maita KC, Torres-Guzman RA, Emam OS, Avila FR, et al. Using AI to detect pain through facial expressions: a review. Bioengineering (Basel). 2023;10(5):548. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhang M, Zhu L, Lin SY, Herr K, Chi CL, Demir I, et al. Using artificial intelligence to improve pain assessment and pain management: a scoping review. J Am Med Inform Assoc. 2023;30(3):570-587. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hughes JD, Chivers P, Hoti K. The clinical suitability of an artificial intelligence-enabled pain assessment tool for use in infants: feasibility and usability evaluation study. J Med Internet Res. 2023;25:e41992. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fang J, Wu W, Liu J, Zhang S. Deep learning-guided postoperative pain assessment in children. Pain. 2023;164(9):2029-2035. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529-536. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sounderajah V, Ashrafian H, Rose S, Shah NH, Ghassemi M, Golub R, et al. A quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies: QUADAS-AI. Nat Med. 2021;27(10):1663-1665. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guo J, Riebler A. meta4diag: Bayesian bivariate meta-analysis of diagnostic test studies for routine practice. J Stat Soft. 2018;83(1):1-31. [ CrossRef ]
• Hammal Z, Cohn JF. Automatic detection of pain intensity. Proc ACM Int Conf Multimodal Interact. 2012;2012:47-52. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Adibuzzaman M, Ostberg C, Ahamed S, Povinelli R, Sindhu B, Love R, et al. Assessment of pain using facial pictures taken with a smartphone. 2015. Presented at: 2015 IEEE 39th Annual Computer Software and Applications Conference; July 01-05, 2015;726-731; Taichung, Taiwan. [ CrossRef ]
• Majumder A, Dutta S, Behera L, Subramanian VK. Shoulder pain intensity recognition using Gaussian mixture models. 2015. Presented at: 2015 IEEE International WIE Conference on Electrical and Computer Engineering (WIECON-ECE); December 19-20, 2015;130-134; Dhaka, Bangladesh. [ CrossRef ]
• Rathee N, Ganotra D. A novel approach for pain intensity detection based on facial feature deformations. J Vis Commun Image Represent. 2015;33:247-254. [ CrossRef ]
• Sikka K, Ahmed AA, Diaz D, Goodwin MS, Craig KD, Bartlett MS, et al. Automated assessment of children's postoperative pain using computer vision. Pediatrics. 2015;136(1):e124-e131. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rathee N, Ganotra D. Multiview distance metric learning on facial feature descriptors for automatic pain intensity detection. Comput Vis Image Und. 2016;147:77-86. [ CrossRef ]
• Zhou J, Hong X, Su F, Zhao G. Recurrent convolutional neural network regression for continuous pain intensity estimation in video. 2016. Presented at: 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); June 26-July 01, 2016; Las Vegas, NV. [ CrossRef ]
• Egede J, Valstar M, Martinez B. Fusing deep learned and hand-crafted features of appearance, shape, and dynamics for automatic pain estimation. 2017. Presented at: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017); May 30-June 03, 2017;689-696; Washington, DC. [ CrossRef ]
• Martinez DL, Rudovic O, Picard R. Personalized automatic estimation of self-reported pain intensity from facial expressions. 2017. Presented at: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); July 21-26, 2017;2318-2327; Honolulu, HI. [ CrossRef ]
• Bourou D, Pampouchidou A, Tsiknakis M, Marias K, Simos P. Video-based pain level assessment: feature selection and inter-subject variability modeling. 2018. Presented at: 2018 41st International Conference on Telecommunications and Signal Processing (TSP); July 04-06, 2018;1-6; Athens, Greece. [ CrossRef ]
• Haque MA, Bautista RB, Noroozi F, Kulkarni K, Laursen C, Irani R. Deep multimodal pain recognition: a database and comparison of spatio-temporal visual modalities. 2018. Presented at: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018); May 15-19, 2018;250-257; Xi'an, China. [ CrossRef ]
• Semwal A, Londhe ND. Automated pain severity detection using convolutional neural network. 2018. Presented at: 2018 International Conference on Computational Techniques, Electronics and Mechanical Systems (CTEMS); December 21-22, 2018;66-70; Belgaum, India. [ CrossRef ]
• Tavakolian M, Hadid A. Deep binary representation of facial expressions: a novel framework for automatic pain intensity recognition. 2018. Presented at: 2018 25th IEEE International Conference on Image Processing (ICIP); October 07-10, 2018;1952-1956; Athens, Greece. [ CrossRef ]
• Tavakolian M, Hadid A. Deep spatiotemporal representation of the face for automatic pain intensity estimation. 2018. Presented at: 2018 24th International Conference on Pattern Recognition (ICPR); August 20-24, 2018;350-354; Beijing, China. [ CrossRef ]
• Wang J, Sun H. Pain intensity estimation using deep spatiotemporal and handcrafted features. IEICE Trans Inf & Syst. 2018;E101.D(6):1572-1580. [ CrossRef ]
• Bargshady G, Soar J, Zhou X, Deo RC, Whittaker F, Wang H. A joint deep neural network model for pain recognition from face. 2019. Presented at: 2019 IEEE 4th International Conference on Computer and Communication Systems (ICCCS); February 23-25, 2019;52-56; Singapore. [ CrossRef ]
• Casti P, Mencattini A, Comes MC, Callari G, Di Giuseppe D, Natoli S, et al. Calibration of vision-based measurement of pain intensity with multiple expert observers. IEEE Trans Instrum Meas. 2019;68(7):2442-2450. [ CrossRef ]
• Lee JS, Wang CW. Facial pain intensity estimation for ICU patient with partial occlusion coming from treatment. 2019. Presented at: BIBE 2019; The Third International Conference on Biological Information and Biomedical Engineering; June 20-22, 2019;1-4; Hangzhou, China.
• Saha AK, Ahsan GMT, Gani MO, Ahamed SI. Personalized pain study platform using evidence-based continuous learning tool. 2019. Presented at: 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC); July 15-19, 2019;490-495; Milwaukee, WI. [ CrossRef ]
• Tavakolian M, Hadid A. A spatiotemporal convolutional neural network for automatic pain intensity estimation from facial dynamics. Int J Comput Vis. 2019;127(10):1413-1425. [ FREE Full text ] [ CrossRef ]
• Bargshady G, Zhou X, Deo RC, Soar J, Whittaker F, Wang H. Ensemble neural network approach detecting pain intensity from facial expressions. Artif Intell Med. 2020;109:101954. [ CrossRef ] [ Medline ]
• Bargshady G, Zhou X, Deo RC, Soar J, Whittaker F, Wang H. Enhanced deep learning algorithm development to detect pain intensity from facial expression images. Expert Syst Appl. 2020;149:113305. [ CrossRef ]
• Dragomir MC, Florea C, Pupezescu V. Automatic subject independent pain intensity estimation using a deep learning approach. 2020. Presented at: 2020 International Conference on e-Health and Bioengineering (EHB); October 29-30, 2020;1-4; Iasi, Romania. [ CrossRef ]
• Huang D, Xia Z, Mwesigye J, Feng X. Pain-attentive network: a deep spatio-temporal attention model for pain estimation. Multimed Tools Appl. 2020;79(37-38):28329-28354. [ CrossRef ]
• Mallol-Ragolta A, Liu S, Cummins N, Schuller B. A curriculum learning approach for pain intensity recognition from facial expressions. 2020. Presented at: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020); November 16-20, 2020;829-833; Buenos Aires, Argentina. [ CrossRef ]
• Peng X, Huang D, Zhang H. Pain intensity recognition via multi‐scale deep network. IET Image Process. 2020;14(8):1645-1652. [ FREE Full text ] [ CrossRef ]
• Tavakolian M, Lopez MB, Liu L. Self-supervised pain intensity estimation from facial videos via statistical spatiotemporal distillation. Pattern Recognit Lett. 2020;140:26-33. [ CrossRef ]
• Xu X, de Sa VR. Exploring multidimensional measurements for pain evaluation using facial action units. 2020. Presented at: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020); November 16-20, 2020;786-792; Buenos Aires, Argentina. [ CrossRef ]
• Pikulkaew K, Boonchieng W, Boonchieng E, Chouvatut V. 2D facial expression and movement of motion for pain identification with deep learning methods. IEEE Access. 2021;9:109903-109914. [ CrossRef ]
• Rezaei S, Moturu A, Zhao S, Prkachin KM, Hadjistavropoulos T, Taati B. Unobtrusive pain monitoring in older adults with dementia using pairwise and contrastive training. IEEE J Biomed Health Inform. 2021;25(5):1450-1462. [ CrossRef ] [ Medline ]
• Semwal A, Londhe ND. S-PANET: a shallow convolutional neural network for pain severity assessment in uncontrolled environment. 2021. Presented at: 2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC); January 27-30, 2021;0800-0806; Las Vegas, NV. [ CrossRef ]
• Semwal A, Londhe ND. ECCNet: an ensemble of compact convolution neural network for pain severity assessment from face images. 2021. Presented at: 2021 11th International Conference on Cloud Computing, Data Science & Engineering (Confluence); January 28-29, 2021;761-766; Noida, India. [ CrossRef ]
• Szczapa B, Daoudi M, Berretti S, Pala P, Del Bimbo A, Hammal Z. Automatic estimation of self-reported pain by interpretable representations of motion dynamics. 2021. Presented at: 2020 25th International Conference on Pattern Recognition (ICPR); January 10-15, 2021;2544-2550; Milan, Italy. [ CrossRef ]
• Ting J, Yang YC, Fu LC, Tsai CL, Huang CH. Distance ordering: a deep supervised metric learning for pain intensity estimation. 2021. Presented at: 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA); December 13-16, 2021;1083-1088; Pasadena, CA. [ CrossRef ]
• Xin X, Li X, Yang S, Lin X, Zheng X. Pain expression assessment based on a locality and identity aware network. IET Image Process. 2021;15(12):2948-2958. [ FREE Full text ] [ CrossRef ]
• Alghamdi T, Alaghband G. Facial expressions based automatic pain assessment system. Appl Sci. 2022;12(13):6423. [ FREE Full text ] [ CrossRef ]
• Barua PD, Baygin N, Dogan S, Baygin M, Arunkumar N, Fujita H, et al. Automated detection of pain levels using deep feature extraction from shutter blinds-based dynamic-sized horizontal patches with facial images. Sci Rep. 2022;12(1):17297. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fontaine D, Vielzeuf V, Genestier P, Limeux P, Santucci-Sivilotto S, Mory E, et al. Artificial intelligence to evaluate postoperative pain based on facial expression recognition. Eur J Pain. 2022;26(6):1282-1291. [ CrossRef ] [ Medline ]
• Hosseini E, Fang R, Zhang R, Chuah CN, Orooji M, Rafatirad S, et al. Convolution neural network for pain intensity assessment from facial expression. 2022. Presented at: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC); July 11-15, 2022;2697-2702; Glasgow, Scotland. [ CrossRef ]
• Huang Y, Qing L, Xu S, Wang L, Peng Y. HybNet: a hybrid network structure for pain intensity estimation. Vis Comput. 2021;38(3):871-882. [ CrossRef ]
• Islamadina R, Saddami K, Oktiana M, Abidin TF, Muharar R, Arnia F. Performance of deep learning benchmark models on thermal imagery of pain through facial expressions. 2022. Presented at: 2022 IEEE International Conference on Communication, Networks and Satellite (COMNETSAT); November 03-05, 2022;374-379; Solo, Indonesia. [ CrossRef ]
• Swetha L, Praiscia A, Juliet S. Pain assessment model using facial recognition. 2022. Presented at: 2022 6th International Conference on Intelligent Computing and Control Systems (ICICCS); May 25-27, 2022;1-5; Madurai, India. [ CrossRef ]
• Wu CL, Liu SF, Yu TL, Shih SJ, Chang CH, Mao SFY, et al. Deep learning-based pain classifier based on the facial expression in critically ill patients. Front Med (Lausanne). 2022;9:851690. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ismail L, Waseem MD. Towards a deep learning pain-level detection deployment at UAE for patient-centric-pain management and diagnosis support: framework and performance evaluation. Procedia Comput Sci. 2023;220:339-347. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vu MT, Beurton-Aimar M. Learning to focus on region-of-interests for pain intensity estimation. 2023. Presented at: 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG); January 05-08, 2023;1-6; Waikoloa Beach, HI. [ CrossRef ]
• Kaur H, Pannu HS, Malhi AK. A systematic review on imbalanced data challenges in machine learning: applications and solutions. ACM Comput Surv. 2019;52(4):1-36. [ CrossRef ]
• Data for meta-analysis of pain assessment from facial images. Figshare. 2023. URL: https:/​/figshare.​com/​articles/​dataset/​Data_for_Meta-Analysis_of_Pain_Assessment_from_Facial_Images/​24531466/​1 [accessed 2024-03-22]

Abbreviations

Edited by A Mavragani; submitted 26.07.23; peer-reviewed by M Arab-Zozani, M Zhang; comments to author 18.09.23; revised version received 08.10.23; accepted 28.02.24; published 12.04.24.

©Jian Huo, Yan Yu, Wei Lin, Anmin Hu, Chaoran Wu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 12.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.