literature review medical device

• Market Access US & International
• Market Access Europe
• MDR / IVDR Consulting
• IVD Admission Strategy
• AI Medical Devices
• And more…
• Quality management systems (ISO 13485)
• ISO 13485 audits
• Quality management representative
• Quality management system as a service
• Software (IEC 62304, FDA)
• Risk Management (ISO 14971)
• Clinical Evaluation
• Performance evaluation of IVDs
• Electrical Safety & IEC 60601
• Human Factors / Usability (IEC 62366 and FDA)
• FDA relevant documents
• Human Factors Research
• Safety and EMC test laboratory
• Biological safety
• Computer System Validation
• Seminar Overview
• Digital Transformation
• Medical Device University
• Post Market Tools
• Our Mission
• Certificates
• Our Customers
• Current Vacancies
• Why the Johner Institute?
• Regulatory Affairs
• The clinical evaluation literature search:…

The clinical evaluation literature search: 6 tips to save you time and stress

The literature search is a key part of the clinical evaluation. It usually involves numerous hours of work. This article will give you six tips to help you efficiently carry out and fully document the literature search.

In the literature search, manufacturers gather together scientific articles, among other reasons, to document the state of the art and provide evidence of the safety, performance, and clinical benefit of their device. But more on this later.

Tip 1: Use the information in the guidelines when searching for literature

There are now several MDCG documents available on the topic of the clinical evaluation, but none that describe how to carry out and document the literature search for the clinical evaluation.

a) MEDDEV 2.7/1 Revision 4 on the literature search

MEDDEV 2.7/1 is also the most important guideline for the literature search under the MDR. The Medical Device Coordination Group says the same.

“For general guidance on a literature search, see MEDDEV 2.7/1 Revision 4, A5. Literature search and literature review protocol, key elements” Section D, MDCG 2020-13

Through the literature search, you find literature on the device under evaluation, the equivalent device and the state of the art, including alternative examination and treatment methods.

Annex 5 of MEDDEV 2.7 /1 Revision 4 describes the most important aspects to remember when documenting the literature search. In it, the guideline requires the objective of the literature search(es) to be documented. Examples of such objectives are:

• Providing data on the device under evaluation (including device name and model)
• Identifying important data for the risk management process (focus on patient population and existing interventions)
• Providing information for the evaluation of the benefit/risk profile
• Giving an overview of the current safety specifications
• Enabling a comparison of possible side effects
• Providing information on benchmark devices

Manufacturers also have to document the search methods .

There will be more on documentation in the second tip.

Additional information can be found in our in-depth article on MEDDEV 2.7/1 .

b) MDCG documents

The MDCG documents do not currently offer any concrete guidance on how the literature search should be carried out. The MDCG document 2020-13 “Clinical evaluation assessment report template” is nevertheless useful:

It is primarily aimed at clinical evaluation reviewers, particularly notified bodies , but it also provides indirect guidance for anyone carrying out a clinical evaluation. Section D deals with literature search and literature review. The requirements listed in this section are the same as the ones in MEDDEV 2.7/1 Revision 4. The focus is on:

• Search categories (e.g., device search or state of the art including clinical condition)
• Scope of the search strategy
• Search and review methods
• Literature search documentation

The MDCG 2020-13 document refers to MEDDEV 2.7/1 Revision 4. So, save some time for the MDCG document and be glad that you can continue working with MEDDEV 2.7/1 Revision 4, especially when it comes to the literature search.

c) Other documents

Other guidance documents on the preparation of the clinical evaluation include, for example, IMDRF MDCE WG/N57FINAL:2019 .

Tip 2: Fully document the literature search

A) literature search protocol.

Incomplete documentation of the literature search will result in a non-conformity. This can lead to unnecessary queries or even deviations in the audit, since the MDCG document 2020-13 explicitly requires notified bodies to review the literature search documentation. It requires reviews to evaluate the following metadata:

• Search terms
• Databases used
• Inclusion and exclusion criteria
• Exclusion of duplicates
• Literature review procedure and documentation
• Search methods

These metadata are incorporated into the literature search protocol.

MDCG 2020-13 requires auditors to pay special attention to the exclusion criteria.

The clinical evaluation should clearly describe the selection criteria with respect to the regulatory purpose to which it will apply. The CER should clearly differentiate between the two types of data (device under evaluation or an equivalent device, state of the art or alternative treatment option). If the data does not relate to either of the above, provide a rationale with respect to its inclusion. Section D, MDCG 2020-13

MDCG 2020-13 requires manufacturers to define and document the selection criteria for the literature searches. The selection criteria should be defined in the context of the clinical evaluation and distinguish between at least two searches for data or information:

• Search for the state of the art
• Search for the device under evaluation/equivalent device

You can find out more on these two searches in tip 3.

b) Additional documentation

The complete documentation of the literature search doesn’t just include the literature search protocol.

The documentation includes all the following documents:

• The aforementioned literature search protocol
• The literature search report, including any deviations from the literature search protocol, and the results of the search
• Complete list of retrieved articles
• Complete list of articles excluded with reasons for exclusion
• Full text copies of relevant documents

Most clinical evaluations that the Johner Institute receives for revision have been rejected by notified bodies because of literature searches where the above were not, or not completely, documented and available.

Tip 3: Remember that there are several literature searches

You are not free to choose what to look for. MEDDEV 2.7/1 Revision 4 requires your literature search to cover at least two essential topics:

• You need the state of the art search to demonstrate the state of the art for your device and to evaluate your device in comparison.
• You need the statements on your own device (or equivalent device) to demonstrate the safety, performance, and clinical benefit of your medical device.

Different search terms are used depending on the objective (see Fig. 1).

You can’t neglect any of these searches. Otherwise there is a risk of a non-conformity in the audit.

The PICO method will help you with literature searches, for example, when trying to find search criteria for the state of the art.

P atient/ P opulation;  I ntervention;  C omparison;  O utcome

The PICO method is used in evidence-based medicine in particular and recommended for use in clinical evaluation literature searches by MEDDEV 2.7/1 Revision 4.

Tip 4: Search in the databases relevant to you

Just as important as the search strategy is choosing which databases you are going to search. The MDR does not give any concrete advice on how to choose the literature databases. However, in article 2(48) it requires peer-reviewed publications.

With a few exceptions, PubMed only contains peer-reviewed publications. In contrast to Embase, PubMed allows free search and registration. However, access to the full texts is not always free of charge on PubMed.

In addition, the MDR mentions “relevant specialist literature” or “databases”. However, the EU regulation leaves it up to the authors of the clinical evaluation to decide which databases they search.

In section D, the MDCG 2020-13 states that multiple databases should be used to minimize bias in the literature review.

MEDDEV 2.7/1 Revision 4, Annex 4 provides some guidance for selecting suitable literature databases. It recommends using MEDLINE, PubMed and other databases such as EMBASE or the Cochrane CENTRAL trials register but does not explicitly require them.

So, you can save costs and start the search in PubMed and use additional databases (e.g., EMBASE) to cover European topics (therapies or medical devices in use in Europe).

A further list of possible sources for literature and clinical data can be found in the article on clinical data .

Tip 5: Use (Boolean) operators

The use of (Boolean) operators allows you to narrow down your literature search, which saves you the trouble of reading non-specific literature sources. However, you can also use Boolean operators to expand the search, especially if you don’t find enough literature sources.

The operators can be used to combine different search terms according to the context. The most well-known Boolean operators are AND, OR and NOT.

• Combining terms with “AND” filters the results for entries that contain all the search terms.
• Combining terms with “OR” filters the results to show entries that contain one of the search terms.
• And combing the terms with “NOT” excludes entries with this search term from your search.

Quotation marks and round brackets are also useful for improving the quality of the search results. Use them to get relevant and specific search results.

• If you put your search term in quotation marks, “”, the search engine will search for the search terms in that exact context and order.
• Round brackets, () can be used to refer a Boolean operator to terms or units.

The following table illustrates this with some examples:

Note that the different databases use different operators. Check the database’s website to find out more. This also affects the interpretation of the search phrase when no brackets are used, as shown in the last example in the previous table. Some databases interpret this search phrase as (“ice pack” AND “reduction of pain”) OR “reduction of edema”.

Tip 6: Don’t read the full text of all the search results

The literature search often returns several hundred publications. At this point, you may wonder whether it is necessary to read the full text of each publication.

You (still) don’t have to read the full texts of all the documents found in the initial search. You can exclude obviously non-relevant publications on the basis of the abstract if they are not related to the clinical evaluation and, for example, don’t have anything to do with the device under evaluation.

However, you should read the full text publication during the literature review at the latest. The MDCG agrees with this view:

“Abstracts lack sufficient detail to allow issues to be evaluated thoroughly and independently, but may be sufficient to allow a first evaluation of the relevance of a paper. Copies of the full text papers and documents should be obtained for the appraisal stage.” Section D, MDCG 2020-13

Summary and conclusion

You should continue to use MEDDEV 2.7/1 Revision 4 as a guide for the literature search. The MDCG 2020-13 document refers to the requirements of MEDDEV 2.7/1 Revision 4 and encourages clinical evaluation reviewers to pay close attention to the following topics:

• Availability of full text publications
• Completeness of the meta-information (e.g., inclusion and exclusion criteria)

Control the range and specificity of the publications found in your literature search by using (Boolean) operators. Save time by excluding obviously irrelevant publications at an early stage, particularly if they do not relate to the state of the art or the device under evaluation.

Remember that in addition to the state of the art, you must at least research the information on equivalent devices. In other words, you must carry out at least two searches.

If you use the tips in this article, you will comply with the MDR’s requirements for literature searches.

Do you have any questions about implementation in specific cases? Email us or send us a contact request . We can review and trim down your documents to ensure conformity so that your clinical evaluation passes the audit.

More Articles

Medical Device Regulation MDR (2017/745) Status 2024 August 27, 2024

Electronic instructions for use for medical devices (EU law) August 22, 2024

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Post comment

  Notify me when new comments are added.

We need your consent before you can continue on our website. If you are under 16 and wish to give consent to optional services, you must ask your legal guardians for permission. We use cookies and other technologies on our website. Some of them are essential, while others help us to improve this website and your experience. Personal data may be processed (e.g. IP addresses), for example for personalized ads and content or ad and content measurement. You can find more information about the use of your data in our privacy policy . You can revoke or adjust your selection at any time under Settings .

• External Media

Individual Privacy Preferences

Cookie Details Privacy Policy Imprint

If you are under 16 and wish to give consent to optional services, you must ask your legal guardians for permission. We use cookies and other technologies on our website. Some of them are essential, while others help us to improve this website and your experience. Personal data may be processed (e.g. IP addresses), for example for personalized ads and content or ad and content measurement. You can find more information about the use of your data in our privacy policy . Here you will find an overview of all cookies used. You can give your consent to whole categories or display further information and select certain cookies.

Accept all Save

Essential cookies enable basic functions and are necessary for the proper function of the website.

Show Cookie Information Hide Cookie Information

Content from video platforms and social media platforms is blocked by default. If External Media cookies are accepted, access to those contents no longer requires manual consent.

Privacy Policy Imprint

Advertisement

A Review of Biomedical Devices: Classification, Regulatory Guidelines, Human Factors, Software as a Medical Device, and Cybersecurity

• Published: 01 August 2023
• Volume 2 , pages 316–341, ( 2024 )

Cite this article

• Felix Tettey   ORCID: orcid.org/0000-0002-2101-0131 1 , 2 ,
• Santosh Kumar Parupelli   ORCID: orcid.org/0000-0003-0535-7618 1 , 2 &
• Salil Desai   ORCID: orcid.org/0000-0002-6116-2105 1 , 2  

2820 Accesses

9 Citations

1 Altmetric

Explore all metrics

Biomedical devices provide a critical role in the healthcare system to positively impact patient well-being. This paper aims to provide the current classifications and subclassifications of hardware and software medical devices according to the Food and Drug Administration (FDA) guidelines. An overview of the FDA regulatory pathway for medical device development such as radiation-emitting electronic product verification, product classification database, Humanitarian Use Device (HUD), premarket approval, premarket notification and clearance, post-market surveillance, and reclassification is provided. Current advances and the advantages of implementing human factors engineering in biomedical device development to reduce the risk of user error, product recalls and effective safe use are discussed. This paper also provides a review of evolving topics such as the Internet of Things (IoT), software as medical devices, artificial intelligence (AI), machine learning (ML), mobile medical devices, and clinical decision software support systems. A comprehensive discussion of the first FDA-approved AI-medical device for the diagnosis of diabetic retinopathy is presented. Further, potential cybersecurity-related risks associated with software-driven AI/ML and IoT medical devices are discussed with an emphasis on government regulations. Futuristic trends in biomedical device development and their implications on patient care and the healthcare system are elucidated.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Copyright from [ 163 ]

Similar content being viewed by others

Software as Medical Device in Europe

The 510(k) Third Party Review Program: Promise and Potential

Explore related subjects.

• Artificial Intelligence

Data Availability

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

U.S. Food and Drugs Administration, Importing Medical Devices (2018), https://www.fda.gov/industry/importing-fda-regulated-products/importing-medical-devices#What is a medical device.

World Health Organization, Medical Devices. https://www.who.int/health-topics/medical-devices#tab=tab_1

C. Peña, K. Li, R. Felten, N. Ogden, M. Melkerson, An example of US Food and Drug Administration device regulation: medical devices indicated for use in acute ischemic stroke. Stroke 38 (6), 1988–1992 (2007)

Article   PubMed   Google Scholar  

S.K. Gupta, Medical device regulations: a current perspective. J. Young Pharm. 8 (1), 6 (2016)

Article   Google Scholar  

S. Desai, S. Parupelli, Additive manufacturing (3D printing), in Maynard's Industrial and Systems Engineering Handbook , 6th ed. (Springer, Berlin, 2022)

S. Bose, S.F. Robertson, A. Bandyopadhyay, Surface modification of biomaterials and biomedical devices using additive manufacturing. Acta Biomater. 66 , 6–22 (2018)

Article   CAS   PubMed   Google Scholar  

M. Olowe, S.K. Parupelli, S. Desai, A review of 3D-printing of microneedles. Pharmaceutics 14 (12), 2693 (2022)

Article   CAS   PubMed   PubMed Central   Google Scholar  

E. Adarkwa, R. Kotoka, S. Desai, 3D printing of polymeric coatings on AZ31 Mg alloy substrate for corrosion protection of biomedical implants. Med. Devices Sensors 4 (1), e10167 (2021)

Article   CAS   Google Scholar  

G. Haeberle, S. Desai, Additive manufacturing (3D printing) of thermoform tooling. Int. J. Mech. Prod. Eng 7 , 1–4 (2019)

Google Scholar  

A. Aljohani, S. Desai, 3D printing of porous scaffolds for medical applications. Am. J. Eng. Appl. Sci. 11 (3), 1076–1085 (2018)

S.K. Parupelli, S. Desai, Understanding hybrid additive manufacturing of functional devices. Am. J. Eng. Appl. Sci. 10 , 264–271 (2017). https://doi.org/10.3844/ajeassp.2017.264.271

F. Aldawood, S. Desai, Additive manufacturing of compensator devices for radiation therapy, in Proceedings of the 2020 IISE Annual Conference (2020)

J. McKenzie, S. Parupelli, D. Martin, S. Desai, Additive manufacturing of multiphase materials for electronics, in IIE Annual Conference. Proceedings , pp. 1133–1138 (2017)

L.J. Kelly, T. Jones, Medical device classification: focus on vascular access. Br. J. Nurs. 27 (14), S14–S19 (2018). https://doi.org/10.12968/bjon.2018.27.14.S14

Centre for Devices & Radiological Health. FDA, Unique device identification and the EHR-types of medical devices and examples (2013), https://slideplayer.com/slide/1480438/ . Accessed 13 July 2022

U.S. Food and Drug Administration-Products and Medical Procedures (2021), https://www.fda.gov/medical-devices/products-and-medical-procedures

A.N. Johnson, Medical Devices & MedTech Products (2020), https://angelanjohnson.com/medicaldevices/

FDA, US Food and Drugs Administration-Classify Your Medical Device (2018), https://www.fda.gov/medical-devices/overview-device-regulation/classify-your-medical-device

US Food and Drug Administration (FDA), Consumers (Medical Device), 2017 . https://www.fda.gov/medical-devices/consumers-medical-devices/learn-if-medical-device-has-been-cleared-fda-marketing

A.V. Kaplan, D.S. Baim, J.J. Smith, D.A. Feigal, M. Simons, D. Jefferys, T.J. Fogarty, R.E. Kuntz, M.B. Leon, Medical device development: from prototype to regulatory approval. Circulation 109 , 3068–3072 (2004)

L.H. Monsein, Primer on medical device regulation. Part II. Regulation of medical devices by the US Food and Drug Administration. Radiology 205 , 10–18 (1997)

L. Keutzer, U.S.H. Simonsson, Medical device apps: an introduction to regulatory affairs for developers. JMIR mHealth uHealth 8 (6), e17567 (2020)

Article   PubMed   PubMed Central   Google Scholar  

B. Zhang, S.B. Shankara, J. Guo, H. Zhang, Pivotal clinical trials with patient-reported outcome measures in premarket approval applications for high-risk medical devices from 2005 to 2018: review, examples, and regulatory considerations. Contemp. Clin. Trials 116 , 106757 (2022)

J. Li, M. Stachowski, Z. Zhang, Application of responsive polymers in implantable medical devices and biosensors, in Switchable and Responsive Surfaces and Materials for Biomedical Applications (Woodhead Publishing, Cambridge, 2015), pp. 259–298

R. Plowman et al., The rate and cost of hospital-acquired infections occurring in patients admitted to selected specialties of a district general hospital in England and the national burden imposed. J. Hosp. Infect. 47 (3), 198–209 (2001). https://doi.org/10.1053/jhin.2000.0881

E.E. Bennett, J. VanBuren, S.L. Bratton, Presence of invasive devices and risks of healthcare-associated infections and sepsis. J. Pediatr. Intensive Care 7 , 188–195 (2018)

N. Buetti et al., Strategies to prevent central line-associated bloodstream infections in acute-care hospitals: 2022 update. Infect. Control Hosp. Epidemiol. 43 (5), 553–569 (2022)

R.B. Wilson, Y. Farooque, Risks and prevention of surgical site infection after hernia mesh repair and the predictive utility of ACS-NSQIP. J. Gastrointest. Surg. 26 (4), 950–964 (2022)

R.J. Pratt et al., epic2: National evidence-based guidelines for preventing healthcare-associated infections in NHS hospitals in England. J. Hosp. Infect. 65 , S1–S59 (2007)

G. Flodgren, L.O. Conterno, A. Mayhew, O. Omar, C.R. Pereira, S. Shepperd, Interventions to improve professional adherence to guidelines for prevention of device‐related infections. Cochrane Database Syst. Rev. (3), CD006559 (2013)

D. Lebeaux, J.-M. Ghigo, C. Beloin, Biofilm-related infections: bridging the gap between clinical management and fundamental aspects of recalcitrance toward antibiotics. Microbiol. Mol. Biol. Rev. 78 (3), 510–543 (2014)

L. Hall-Stoodley et al., Towards diagnostic guidelines for biofilm-associated infections. FEMS Immunol. Med. Microbiol. 65 (2), 127–145 (2012)

M.K. Kasliwal, L.A. Tan, V.C. Traynelis, Infection with spinal instrumentation: review of pathogenesis, diagnosis, prevention, and management. Surg. Neurol. Int. 4 (Suppl 5), S392 (2013)

PubMed   PubMed Central   Google Scholar  

L. Sangkum, G.L. Liu, L. Yu, H. Yan, A.D. Kaye, H. Liu, Minimally invasive or noninvasive cardiac output measurement: an update. J. Anesth. 30 (3), 461–480 (2016). https://doi.org/10.1007/s00540-016-2154-9

C. Lin, Y. Tsai, Y. Lu, J. Yang, M. Chen, Application of a novel biosensor for salivary conductivity in detecting chronic kidney disease. Biosensors 12 (3), 178 (2022). https://doi.org/10.3390/bios120301782022

W. Li, O. Auciello, R.N. Premnath, B. Kabius, Giant dielectric constant dominated by Maxwell–Wagner relaxation in Al 2 O 3 /TiO 2 nanolaminates synthesized by atomic layer deposition. Appl. Phys. Lett. 96 (16), 162907 (2010)

Article   ADS   Google Scholar  

R. Magjarević, Non-invasive and minimally invasive medical devices. Report, May, pp. 10–12 (2017)

C.H. Chen, T.H. Tao, Y.H. Chou, Y.W. Chuang, T.B. Chen, Arteriovenous fistula flow dysfunction surveillance: early detection using pulse radar sensor and machine learning classification. Biosensors (2021). https://doi.org/10.3390/bios11090297

J.F. Mooney et al., Relative value of cystatin C and creatinine-based estimates of glomerular filtration rate in predicting long-term mortality after cardiac surgery: a cohort study. BMJ Open 9 (9), e029379 (2019)

T.K. Chen, D.H. Knicely, M.E. Grams, Chronic kidney disease diagnosis and management: a review. JAMA 322 (13), 1294–1304 (2019). https://doi.org/10.1001/jama.2019.14745

K.P. Bhatia et al., Consensus statement on the classification of tremors from the task force on tremor of the International Parkinson and Movement Disorder Society. Mov. Disord. 33 (1), 75–87 (2018)

Article   MathSciNet   PubMed   Google Scholar  

E.D. Louis, Tremor. Contin. Lifelong Learn. Neurol. 25 (4), 959–975 (2019)

V. Srinivasan, V.K. Pamula, M.G. Pollack, R.B. Fair, Clinical diagnostics on human whole blood, plasma, serum, urine, saliva, sweat, and tears on a digital microfluidic platform, in Proc. MicroTAS , pp. 1287–1290 (2003)

G. Gaobotse, E. Mbunge, J. Batani, B. Muchemwa, Non-invasive smart implants in healthcare: Redefining healthcare services delivery through sensors and emerging digital health technologies. Sensors Int. 3 , 100156 (2022). https://doi.org/10.1016/j.sintl.2022.100156

K. Dixit, S. Fardindoost, A. Ravishankara, N. Tasnim, M. Hoorfar, Exhaled breath analysis for diabetes diagnosis and monitoring: relevance, challenges and possibilities. Biosensors 11 (12), 476 (2021)

K. Lee et al., Real-world outcomes of glucose sensor use in type 1 diabetes—findings from a large UK centre. Biosensors 11 (11), 457 (2021)

Graphene patches over diabetes treatment (2016), https://www.sciencedirect.com/science/article/abs/pii/S1748013216301347?via%3Dihub

F.F. Franco, R.A. Hogg, L. Manjakkal, Cu 2 O-based electrochemical biosensor for non-invasive and portable glucose detection. Biosensors (2022). https://doi.org/10.3390/bios12030174

J. Kim, A.S. Campbell, B.E.-F. de Avila, J. Wang, Wearable biosensors for healthcare monitoring. Nat. Biotechnol. 37 (4), 389–406 (2019)

R.J. Morrison et al., Regulatory considerations in the design and manufacturing of implantable 3D-printed medical devices. Clin. Transl. Sci. 8 (5), 594–600 (2015). https://doi.org/10.1111/cts.12315

E. Adarkwa, A. Roy, J. Ohodnicki, B. Lee, P.N. Kumta, S. Desai, 3D printing of drug-eluting bioactive multifunctional coatings for orthopedic applications. Int. J. Bioprinting 9 (2), 661 (2023)

S.K. Parupelli, S. Desai, Hybrid additive manufacturing (3D printing) and characterization of functionally gradient materials via in situ laser curing. Int. J. Adv. Manuf. Technol. 110 , 543–556 (2020)

F.K. Aldawood, S.X. Chang, S. Desai, Design and manufacture of a high precision personalized electron bolus device for radiation therapy. Med. Devices Sensors 3 (6), e10077 (2020)

G. Haeberle, S. Desai, Investigating rapid thermoform tooling via additive manufacturing (3D printing). Am. J. Appl. Sci 16 , 238–243 (2019)

J. McKenzie, S. Desai, Investigating sintering mechanisms for additive manufacturing of conductive traces. Am. J. Eng. Appl. Sci. 11 (2), 652–662 (2018)

S. Desai, B. Bidanda, P.J. Bártolo, Emerging trends in the applications of metallic and ceramic biomaterials, in Bio-Materials and Prototyping Applications in Medicine (Springer, Cham, 2021), pp. 1–17

S. Desai, M.R. Shankar, Emerging trends in polymers, composites, and nano biomaterial applications, in Bio-Materials and Prototyping Applications in Medicine (Springer, Cham, 2021), pp. 19–34

S. Desai, P. Gomes, Design for nano/micro manufacturing using a flexible decision making technique (AHP). J. Udyog Pragati 39 (2), 18–25 (2015)

J. Perkins et al., Direct writing of polymeric coatings on magnesium alloy for tracheal stent applications. Ann. Biomed. Eng. 43 , 1158–1165 (2015)

S. Desai, B. Harrison, Direct-writing of biomedia for drug delivery and tissue regeneration, in Printed Biomaterials: Novel Processing and Modeling Techniques for Medicine and Surgery , pp. 71–89 (2010)

S. Desai, M.R. Shankar, Polymers, composites and nano biomaterials: current and future developments, in Bio-Materials and Prototyping Applications in Medicine (Springer, Cham, 2008), pp. 15–26

S. Desai, B. Bidanda, Metallic and ceramic biomaterials: current and future developments, in Bio-Materials and Prototyping Applications in Medicine (Springer, Cham, 2008), pp. 1–14

P. Honigmann, N. Sharma, B. Okolo, U. Popp, B. Msallem, F.M. Thieringer, Patient-specific surgical implants made of 3D printed PEEK: material, technology, and scope of surgical application. Biomed. Res. Int. 2018 , 4520636 (2018)

U.S. Food and Drugs Administration, Medical applications of 3D printing (2017), https://www.fda.gov/medical-devices/3d-printing-medical-devices/medical-applications-3d-printing

U.S. Food and Drugs Administration, Technical considerations for additive manufactured medical devices (2018), https://www.fda.gov/regulatory-information/search-fda-guidance-documents/technical-considerations-additive-manufactured-medical-devices

A.A. Raheem et al., A review on development of bio-inspired implants using 3D printing. Biomimetics 6 (4), 65 (2021)

Article   MathSciNet   PubMed   PubMed Central   Google Scholar  

A.L. Jardini et al., Improvement in cranioplasty: advanced prosthesis biomanufacturing. Procedia CIRP 49 , 203–208 (2016)

X. Chen, J.K. Possel, C. Wacongne, A.F. Van Ham, P.C. Klink, P.R. Roelfsema, 3D printing and modelling of customized implants and surgical guides for non-human primates. J. Neurosci. Methods 286 , 38–55 (2017)

K. Phan, A. Sgro, M.M. Maharaj, P. D’Urso, R.J. Mobbs, Application of a 3D custom printed patient specific spinal implant for C1/2 arthrodesis. J. Spine Surg. 2 (4), 314 (2016)

A. Dzian, J. Živčák, R. Penciak, R. Hudák, Implantation of a 3D-printed titanium sternum in a patient with a sternal tumor. World J. Surg. Oncol. 16 , 1–4 (2018)

S. Gerke, B. Babic, T. Evgeniou, I.G. Cohen, The need for a system view to regulate artificial intelligence/machine learning-based software as medical device. npj Digit. Med. 3 (1), 1–4 (2020). https://doi.org/10.1038/s41746-020-0262-2

S. Desai, M. Lovell, CFD analysis of a continuous inkjet print head for direct write fabrication, in ASME International Mechanical Engineering Congress and Exposition , vol. 43076, pp. 209–213 (2007)

J. Cordeiro, S. Desai, Process parameter studies of molecular dynamics models to control substrate wettability, in International Manufacturing Science and Engineering Conference , vol. 56826, p. V001T02A025 (2015)

J. Cordeiro, S. Desai, Exploring nano scale design space with molecular dynamics simulations, in IIE Annual Conference Proceedings , p. 856 (2015)

S. Desai, Methods and Apparatus for Manufacturing Micro-and/or Nano-Scale Features. Google Patents, 28 Nov 2013

S. Desai, M. Lovell, Modeling fluid–structure interaction in a direct write manufacturing process. J. Mater. Process. Technol. 212 (10), 2031–2040 (2012)

E. Adarkwa, S. Desai, Scalable droplet based manufacturing using in-flight laser evaporation. J. Nanoeng. Nanomanuf. 6 (2), 87–92 (2016)

S. Desai, M. Lovell, Multiphysics modeling of a piezoelectric bimorph disc in a direct write fabrication process, in ASME International Mechanical Engineering Congress and Exposition , vol. 42347, pp. 437–442 (2005)

S. Desai, M. Lovell, Coupled field analysis of a piezoelectric bimorph disc within a CIJ microfabrication process, in IIE Annual Conference. Proceedings , p. 1 (2006)

S. Murgu, H. Colt, Tracheobronchomalacia and excessive dynamic airway collapse. Clin. Chest Med. 34 (3), 527–555 (2013)

S. Desai, J. Perkins, B.S. Harrison, J. Sankar, Understanding release kinetics of biopolymer drug delivery microcapsules for biomedical applications. Mater. Sci. Eng. B 168 (1–3), 127–131 (2010)

S.K. Parupelli, A. Aljohani, S. Desai, S. Khanal, N. Bhattarai, Direct jet printing and characterization of calcium alginate microcapsules for biomedical applications, in IIE Annual Conference. Proceedings , pp. 300–305 (2019)

S. Desai, A. Moore, B. Harrison, J. Sankar, Understanding microdroplet formations for biomedical applications, in ASME International Mechanical Engineering Congress and Exposition , vol. 48760, pp. 119–123 (2008)

S. Desai, J. Sankar, A. Moore, B. Harrison, Biomanufacturing of microcapsules for drug delivery and tissue engineering applications, in IIE Annual Conference. Proceedings , p. 507 (2008)

J.L. Perkins, S. Desai, B. Harrison, J. Sankar, Understanding release kinetics of calcium alginate microcapsules using drop on demand inkjet printing, in ASME International Mechanical Engineering Congress and Exposition , vol. 43871, pp. 77–82 (2009)

S. Bose, K.D. Traxel, A.A. Vu, A. Bandyopadhyay, Clinical significance of three-dimensional printed biomaterials and biomedical devices. MRS Bull. 44 (6), 494–504 (2019)

Article   PubMed   PubMed Central   ADS   Google Scholar  

M. Zarek, N. Mansour, S. Shapira, D. Cohn, 4D printing of shape memory-based personalized endoluminal medical devices. Macromol. Rapid Commun. 38 (2), 1–6 (2017). https://doi.org/10.1002/marc.201600628

J. Li, C. Wu, P.K. Chu, M. Gelinsky, 3D printing of hydrogels: Rational design strategies and emerging biomedical applications. Mater. Sci. Eng. R Rep. 140 , 100543 (2020)

A. Kirillova, R. Maxson, G. Stoychev, C.T. Gomillion, L. Ionov, 4D biofabrication using shape-morphing hydrogels. Adv. Mater. 29 (46), 1703443 (2017)

H. Wei, Q. Zhang, Y. Yao, L. Liu, Y. Liu, J. Leng, Direct-write fabrication of 4D active shape-changing structures based on a shape memory polymer and its nanocomposite. ACS Appl. Mater. Interfaces 9 (1), 876–883 (2017)

M.S. Cabrera, B. Sanders, O.J.G.M. Goor, A. Driessen-Mol, C.W.J. Oomens, F.P.T. Baaijens, Computationally designed 3D printed self-expandable polymer stents with biodegradation capacity for minimally invasive heart valve implantation: a proof-of-concept study. 3D Print. Addit. Manuf. 4 (1), 19–29 (2017)

Y. Wang, H. Cui, T. Esworthy, D. Mei, Y. Wang, L.G. Zhang, Emerging 4D printing strategies for next-generation tissue regeneration and medical devices. Adv. Mater. 34 (20), 2109198 (2022)

U.S. Food and Drugs Administration, Laser Products and Instruments (2023), https://www.fda.gov/radiation-emitting-products/home-business-and-entertainment-products/laser-products-and-instruments .

US Food and Drugs Administration, Radiation-Emitting Electronic Products (2018), https://www.fda.gov/industry/regulated-products/radiation-emitting-electronic-products . Accessed 8 Jul 2022

U.S. Food and Drugs Administration, Device Classification Panels (2018), https://www.fda.gov/medical-devices/classify-your-medical-device/device-classification-panels .

U.S. Food and Drugs Administration, Product Code Classification Database (2018), https://www.fda.gov/medical-devices/classify-your-medical-device/product-code-classification-database .

US Food and Drug Administration, Premarket Notification 510(k), March 13 (2020), https://www.fda.gov/medical-devices/premarket-submissions-selecting-and-preparing-correct-submission/premarket-notification-510k . Accessed 8 Jul 2022

US Food and Drug Administration, Postmarket Requirements (Devices) (2018), https://www.fda.gov/medical-devices/device-advice-comprehensive-regulatory-assistance/postmarket-requirements-devices

U.S. Food and Drugs Administration, Reclassification (2021), https://www.fda.gov/about-fda/cdrh-transparency/reclassification

U.S. Food and Drugs Administration, Medical Device Accessories (2020), https://www.fda.gov/medical-devices/classify-your-medical-device/medical-device-accessories

U.S. Food and Drugs Administration, CFR—Code of Federal Regulations Title 21 (2023), https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/cfrsearch.cfm?fr=820.3

W. Gilmore, The User-Computer Interface in Process Control: A Human Factors Engineering Handbook (Elsevier, Amsterdam, 2012)

R. Gagnon et al., A user-centered evaluation of three intravenous infusion pumps, in Proceedings of the Human Factors and Ergonomics Society Annual Meeting , vol. 48, no. 15, pp. 1773–1777 (2004)

L. Lin, K.J. Vicente, D.J. Doyle, Patient safety, potential adverse drug events, and medical device design: a human factors engineering approach. J. Biomed. Inform. 34 (4), 274–284 (2001)

D. Sawyer, K.J. Aziz, C.L. Backinger, E.T. Beers, A. Lowery, S.M. Sykes, An introduction to human factors in medical devices, U.S. Department of Health and Human Services Public Health Service Food and Drug Administration Center for Devices and Radiological Health, p. 55 (1996)

N.E. Schaeffer, The role of human factors in the design and development of an insulin pump. J. Diabetes Sci. Technol. 6 (2), 260–264 (2012). https://doi.org/10.1177/193229681200600208

J.-E. Kim, L. Kessler, Z. McCauley, I. Niiyama, L.N. Boyle, Human factors considerations in designing a personalized mobile dialysis device: an interview study. Appl. Ergon. 85 , 103003 (2020)

R. North, C. Pospisil, R.J. Clukey, C.G. Parkin, Impact of human factors testing on medical device design: validation of an automated CGM sensor applicator. J. Diabetes Sci. Technol. 13 (5), 949–953 (2019). https://doi.org/10.1177/1932296819831071

A.L. Cassano, Applying human factors to the procurement of electrosurgical medical devices: a case study. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 47 (15), 1815–1819 (2003)

C.J. Vincent, Y. Li, A. Blandford, Integration of human factors and ergonomics during medical device design and development: it’s all about communication. Appl. Ergon. 45 (3), 413–419 (2014). https://doi.org/10.1016/j.apergo.2013.05.009

G. Ginsburg, Human factors engineering: a tool for medical device evaluation in hospital procurement decision-making. J. Biomed. Inform. 38 (3), 213–219 (2005). https://doi.org/10.1016/j.jbi.2004.11.008

J. Nielsen, R. Molich, Heuristic evaluation of user interfaces, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , pp. 249–256 (1990)

B. Schneiderman, Designing the User Interface. Strategies for Effective Human–Computer Interaction (Addison-Wesley, Hoboken, 1992)

M.-H. Maras, Internet of Things: security and privacy implications. Int. Data Priv. Law 5 (2), 99 (2015)

A. Chacko, T. Hayajneh, Security and privacy issues with IoT in healthcare. EAI Endorsed Trans. Pervasive Heal. Technol. 4 (14), 1–8 (2018). https://doi.org/10.4108/eai.13-7-2018.155079

F.Z. Amara, M. Hemam, M. Djezzar, M. Maimor, Semantic web and Internet of Things: challenges, applications and perspectives. J. ICT Stand. 10 (2), 261–292 (2022)

B. Pradhan, S. Bhattacharyya, K. Pal, IoT-based applications in healthcare devices. J. Healthc. Eng. 2021 , 6632599 (2021). https://doi.org/10.1155/2021/6632599

L.M. Dang, M.J. Piran, D. Han, K. Min, H. Moon, A survey on internet of things and cloud computing for healthcare. Electronics 8 (7), 768 (2019)

R. Hughes, Patient Safety and Quality: An Evidence-Based Handbook for Nurses (Agency for Healthcare Research and Quality, Rockville, 2008)

S.E. Bibri, Ethical implications of AMI and the IoT: risks to privacy, security, and trust, and prospective technological safeguards, in The Shaping of Ambient Intelligence and the Internet of Things (Atlantis Press, Paris, 2015), pp. 217–238

D.S. Punithavathani, K. Sujatha, J.M. Jain, Surveillance of anomaly and misuse in critical networks to counter insider threats using computational intelligence. Cluster Comput. 18 (1), 435–451 (2015)

B. Altubaishe, S. Desai, Multicriteria decision making in supply chain management using FMEA and hybrid AHP-PROMETHEE algorithms. Sensors 23 (8), 4041 (2023)

U.S. Food and Drugs Administration, Software as a medical device (SaMD) (2018), https://www.fda.gov/medical-devices/digital-health-center-excellence/software-medical-device-samd

US Food and Drugs Administration, Policy for device software functions and mobile medical applications. Guidance for Industry and Food and Drug Administration Staff (2019), https://www.fda.gov/media/80958/download

U.S. Food and Drugs Administration, Digital Health Criteria (2018), https://www.fda.gov/medical-devices/digital-health-center-excellence/digital-health-criteria

International Medical Device Regulators Forum, Software as a medical device: possible framework for risk categorization and corresponding considerations (2014), http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-140918-samd-framework-risk-categorization-141013.pdf

US Food and Drugs Administration, What are examples of Software as a Medical Device? (2017) https://www.fda.gov/medical-devices/software-medical-device-samd/what-are-examples-software-medical-device

Y. Fang, D. Liu, Z. Jiang, H. Wang, Monitoring of sleep breathing states based on audio sensor utilizing Mel-Scale features in home healthcare. J. Healthc. Eng. 2023 , 6197564 (2023). https://doi.org/10.1155/2023/6197564

E.F. Camargos, F.M. Louzada, O.T. Nóbrega, Wrist actigraphy for measuring sleep in intervention studies with Alzheimer’s disease patients: application, usefulness, and challenges. Sleep Med. Rev. 17 (6), 475–488 (2013)

S.-J. Zhou et al., Measuring sleep stages and screening for obstructive sleep apnea with a wearable multi-sensor system in comparison to polysomnography. Nat. Sci. Sleep 15 , 353–362 (2023). https://doi.org/10.2147/NSS.S406359

V. Patel, A. Chesmore, C.M. Legner, S. Pandey, Trends in workplace wearable technologies and connected-worker solutions for next-generation occupational safety, health, and productivity. Adv. Intell. Syst. 4 (1), 2100099 (2022)

Y. Xue, A review on intelligent wearables: Uses and risks. Hum. Behav. Emerg. Technol. 1 (4), 287–294 (2019)

M.C. Schall Jr., R.F. Sesek, L.A. Cavuoto, Barriers to the adoption of wearable sensors in the workplace: a survey of occupational safety and health professionals. Hum. Factors 60 (3), 351–362 (2018)

J. Kim et al., Simultaneous monitoring of sweat and interstitial fluid using a single wearable biosensor platform. Adv. Sci. 5 (10), 1800880 (2018)

A. Koh et al., A soft, wearable microfluidic device for the capture, storage, and colorimetric sensing of sweat. Sci. Transl. Med. 8 (366), 1-366ra165 (2016)

J. Kim et al., Non-invasive mouthguard biosensor for continuous salivary monitoring of metabolites. Analyst 139 (7), 1632–1636 (2014)

Article   CAS   PubMed   ADS   Google Scholar  

F. Güder et al., Paper-based electrical respiration sensor. Angew. Chem. Int. Ed. 55 (19), 5727–5732 (2016)

M. Padash, C. Enz, S. Carrara, Microfluidics by additive manufacturing for wearable biosensors: a review. Sensors 20 (15), 4236 (2020)

Article   CAS   PubMed   PubMed Central   ADS   Google Scholar  

U.S. Food and Drugs Administration, Wireless Medical Device (2018), https://www.fda.gov/medical-devices/digital-health-center-excellence/wireless-medical-devices

P. Hamet, J. Tremblay, Artificial intelligence in medicine. Metabolism 69 , S36–S40 (2017). https://doi.org/10.1016/j.metabol.2017.01.011

L. Monostori, in Artificial Intelligence BT, in CIRP Encyclopedia of Production Engineering , ed. by L. Laperrière, G. Reinhart (Springer, Berlin, 2014), pp. 47–50

F. Jiang et al., Artificial intelligence in healthcare: past, present and future. Stroke Vasc. Neurol. 2 (4), 230–243 (2017)

A.M. Darcy, A.K. Louie, L.W. Roberts, Machine learning and the profession of medicine. JAMA 315 (6), 551–552 (2016)

H.J. Murff et al., Automated identification of postoperative complications within an electronic medical record using natural language processing. JAMA 306 (8), 848–855 (2011)

U.S. Food and Drugs Administration, Artificial Intelligence and Machine Learning in Software as a Medical Device (2021), https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device

Y. Xu et al., Artificial intelligence: a powerful paradigm for scientific research. Innovations 2 (4), 100179 (2021). https://doi.org/10.1016/j.xinn.2021.100179

Article   MathSciNet   Google Scholar  

M. Ogunsanya, S. Desai, Predictive modeling of additive manufacturing process using deep learning algorithm, in IIE Annual Conference. Proceedings , pp. 1–6 (2022)

H. Elhoone, T. Zhang, M. Anwar, S. Desai, Cyber-based design for additive manufacturing using artificial neural networks for Industry 4.0. Int. J. Prod. Res. 58 (9), 2841–2861 (2020)

T. Akter, S. Desai, Developing a predictive model for nanoimprint lithography using artificial neural networks. Mater. Des. 160 , 836–848 (2018)

S. Desai, C. Dean, Y. Desai, Cyber-enabled concurrent material and process selection in a flexible design for manufacture paradigm. Int. J. Adv. Manuf. Technol. 97 (5–8), 1719–1731 (2018)

K.W. Johnson et al., Artificial intelligence in cardiology. J. Am. Coll. Cardiol. 71 (23), 2668–2679 (2018). https://doi.org/10.1016/j.jacc.2018.03.521

A.-M. Singeap, C. Stanciu, A. Trifan, Capsule endoscopy: the road ahead. World J. Gastroenterol. 22 (1), 369 (2016)

B. Sushma, P. Aparna, Recent developments in wireless capsule endoscopy imaging: Compression and summarization techniques. Comput. Biol. Med. 149 , 106087 (2022). https://doi.org/10.1016/j.compbiomed.2022.106087

D. Nikolayev, M. Zhadobov, R. Sauleau, P. Karban, Antennas for ingestible capsule telemetry, in Advances in Body-Centric Wireless Communication: Applications and State-of-the-Art (IET, London, 2016), pp. 143–186

P.P. Stanich, B. Kleinman, K. Betkerur, N. Mehta Oza, K. Porter, M.M. Meyer, Video capsule endoscopy is successful and effective in outpatients with implantable cardiac devices. Dig. Endosc. 26 (6), 726–730 (2014)

I. Tziortziotis, F.-M. Laskaratos, S. Coda, Role of artificial intelligence in video capsule endoscopy. Diagnostics 11 (7), 1192 (2021)

G.S. Raju, L. Gerson, A. Das, B. Lewis, American Gastroenterological Association (AGA) Institute technical review on obscure gastrointestinal bleeding. Gastroenterology 133 (5), 1697–1717 (2007)

H. Bay, A. Ess, T. Tuytelaars, L. Van Gool, Speeded-Up Robust Features (SURF). Comput. Vis. Image Underst. 110 (3), 346–359 (2008). https://doi.org/10.1016/j.cviu.2007.09.014

D.K. Iakovidis, S. Tsevas, A. Polydorou, Reduction of capsule endoscopy reading times by unsupervised image mining. Comput. Med. Imaging Graph. 34 (6), 471–478 (2010). https://doi.org/10.1016/j.compmedimag.2009.11.005

H.-G. Lee, M.-K. Choi, B.-S. Shin, S.-C. Lee, Reducing redundancy in wireless capsule endoscopy videos. Comput. Biol. Med. 43 (6), 670–682 (2013). https://doi.org/10.1016/j.compbiomed.2013.02.009

S.C. Payne, J.B. Furness, M.J. Stebbing, Bioelectric neuromodulation for gastrointestinal disorders: effectiveness and mechanisms. Nat. Rev. Gastroenterol. Hepatol. 16 (2), 89–105 (2019)

D. Miley, L.B. Machado, C. Condo, A.E. Jergens, K.-J. Yoon, S. Pandey, Video capsule endoscopy and ingestible electronics: emerging trends in sensors, circuits, materials, telemetry, optics, and rapid reading software. Adv. Devices Instrum. (2021). https://doi.org/10.34133/2021/9854040

T. Nakamura, Capsule endoscopy in Japan. Dig. Endosc. 34 , 76–78 (2022)

J. Park et al., Recent development of computer vision technology to improve capsule endoscopy. Clin. Endosc. 52 (4), 328–333 (2019)

X. Jia, M.Q.-H. Meng, Gastrointestinal bleeding detection in wireless capsule endoscopy images using handcrafted and CNN features, in 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) , pp. 3154–3157 (2017). https://doi.org/10.1109/EMBC.2017.8037526

Y. Yuan, M.Q. Meng, Deep learning for polyp recognition in wireless capsule endoscopy images. Med. Phys. 44 (4), 1379–1389 (2017)

D.K. Iakovidis, A. Koulaouzidis, Software for enhanced video capsule endoscopy: challenges for essential progress. Nat. Rev. Gastroenterol. Hepatol. 12 (3), 172–186 (2015). https://doi.org/10.1038/nrgastro.2015.13

Z. Ding et al., Gastroenterologist-level identification of small-bowel diseases and normal variants by capsule endoscopy using a deep-learning model. Gastroenterology 157 (4), 1044–1054 (2019)

T. Aoki et al., Clinical usefulness of a deep learning-based system as the first screening on small-bowel capsule endoscopy reading. Dig. Endosc. 32 (4), 585–591 (2020)

G. Ciuti et al., Frontiers of robotic endoscopic capsules: a review. J. Micro-bio Robot. 11 , 1–18 (2016)

J.C. Norton et al., Intelligent magnetic manipulation for gastrointestinal ultrasound. Sci. Robot. 4 (31), eaav7725 (2019)

J. Min, Y. Yang, Z. Wu, W. Gao, Robotics in the gut. Adv. Ther. 3 (4), 1900125 (2020)

U.S. Food and Drugs Administration, Ingestible telemetric gastrointestinal capsule imaging system—final class ii special controls guidance document for industry and FDA (2018), https://www.fda.gov/medical-devices/guidance-documents-medical-devices-and-radiation-emitting-products/ingestible-telemetric-gastrointestinal-capsule-imaging-system-final-class-ii-special-controls

B. Prabhakar, R.K. Singh, K.S. Yadav, Artificial intelligence (AI) impacting diagnosis of glaucoma and understanding the regulatory aspects of AI-based software as medical device. Comput. Med. Imaging Graph. 87 , 101818 (2021). https://doi.org/10.1016/j.compmedimag.2020.101818

X. Zhang et al., Prevalence of diabetic retinopathy in the United States, 2005–2008. JAMA 304 (6), 649–656 (2010)

Cruise Ophthalmic, Portable eye fundus camera Digital Retinal Scan Photography. https://www.aliexpress.com/item/3256801216401596.html?gatewayAdapt=4itemAdapt

Digital Diagnosis-AI the right way. https://www.digitaldiagnostics.com/products/eye-disease/idx-dr/?gclid=CjwKCAjw46CVBhB1EiwAgy6M4hNGGgKBv1kg0j_9nkFxQHp5UqBclhg-5WyP8KGTFVOLrpYWTGp7fRoCuPIQAvD_BwE

F. Mahgoub, Here’s How the FDA Approval of IDx-DR Will Impact Patients (2018), https://introwellness.com/eyes/idx-dr/

E.H. Nepal, Review of 3nethra Classic Digital Non-mydriatic Fundus Camera (2021), https://www.eyehealthnepal.com/3nethra-classic-fundus-camera/

L. Tsang et al., The impact of artificial intelligence on medical innovation in the European Union and United States. Intellect. Prop. Technol. Law J. 29 (8), 3–12 (2017)

U.S. Food and Drugs Administration, Device Software Functions Including Mobile Medical Applications (2022), https://www.fda.gov/medical-devices/digital-health-center-excellence/device-software-functions-including-mobile-medical-applications

U.S. Food and Drugs Administration, Medical Device Data Systems (2019), https://www.fda.gov/medical-devices/general-hospital-devices-and-supplies/medical-device-data-systems

G. Freckmann et al., Insulin pump therapy for patients with type 2 diabetes mellitus: evidence, current barriers, and new technologies. J. Diabetes Sci. Technol. 15 (4), 901–915 (2021)

N.J. Wimmer et al., Assessing the cost burden of United States FDA-mandated post-approval studies for medical devices. J. Health Care Finance 2016 (Spec FEATURES) (2016)

Download references

Acknowledgements

The authors acknowledge the funding from the National Science Foundation (NSF) #1663128, #2100739, #2100850, #2200538. We would like to thank the Center of Excellence in Product Design and Advanced Manufacturing (CEPDAM), North Carolina A & T State University.

The authors acknowledge the funding from the National Science Foundation (NSF) #1663128, #2100739, #2100850, #2200538.

Author information

Authors and affiliations.

Industrial and Systems Engineering Department, North Carolina Agricultural and Technical State University, Greensboro, NC, 27411, USA

Felix Tettey, Santosh Kumar Parupelli & Salil Desai

Center of Excellence in Product Design and Advanced Manufacturing, North Carolina Agricultural and Technical State University, Greensboro, NC, 27411, USA

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to the composition of the manuscript. The final version of the manuscript has been approved by all authors.

Corresponding author

Correspondence to Salil Desai .

Ethics declarations

Competing interests.

There are no competing interests for this review article.

Ethical Approval

All the information presented is cited appropriately and relevant literature in support of the claims made.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Tettey, F., Parupelli, S.K. & Desai, S. A Review of Biomedical Devices: Classification, Regulatory Guidelines, Human Factors, Software as a Medical Device, and Cybersecurity. Biomedical Materials & Devices 2 , 316–341 (2024). https://doi.org/10.1007/s44174-023-00113-9

Download citation

Received : 26 May 2023

Accepted : 29 June 2023

Published : 01 August 2023

Issue Date : March 2024

DOI : https://doi.org/10.1007/s44174-023-00113-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Cybersecurity
• Human factors
• Software as a Medical Device (SaMD)
• Find a journal
• Publish with us
• Track your research

Literature searches and reviews for medical devices - what to know before you start

• By Sandra Gopinath
• Chief Regulatory Officer
• Share on LinkedIn
• Share on Facebook
• Share by email

Would you be surprised to know that the word ‘literature’ pops up in MedDev 2.7/1 rev 4 approximately 100 times? For example, if you work on Clinical Evaluation Reports (CERs) , you are probably starting to realise how crucial literature searches & reviews are in achieving regulatory compliance under the new MDR.

The importance of critical appraisal of available evidence through a systematic literature review is emphasised repeatedly in MedDev 2.7/1 rev 4 and is crucial in ensuring compliance with the new EU MDR . In this article, we discuss what you should know before undertaking a medical device literature search & review for Clinical Evaluation , Risk Management or other purposes under the MDR.

1. Start with research questions

Research questions are intended to guide search term selection, helping to frame the boundaries of the literature search. In short, research questions set out what you intend to discover from the literature review .

MedDev 2.7/1 rev 4 recommends using a PICO framework to help you in perfecting your research questions. PICO format is extensively employed in evidence-based clinical practice. This approach results in a “well-built” query that specifies four concepts:

• the Patient issue or Population
• the Intervention
• the Comparison (if one exists)
• the Outcome(s)

Once you formulate your research questions, accordingly, determine the information required to answer these questions. This leads us to tip number 2.

2. Choose your search terms

Now that you have the research questions and the information required to answer these queries, convert the questions into searchable terms .

While formulating your search terms, you can use Boolean operators such as ‘OR,’ ‘AND,’ and ‘NOT’ to tailor the results according to your requirements.

The search terms are then applied to scientific literature databases such as PubMed, MEDLINE, Google scholar, EMBASE and Cochrane review as sources to identify the evidence. A search in these databases can generate an overwhelmingly high number of results.

Analysing all of them to identify the appropriate ones is going to be a ‘Herculean task’ if you don’t choose search terms carefully and have an efficient exclusion strategy.

3. Dealing with exclusions

Video 1: Five common pitfalls when writing a Clinical Evaluation Report

To separate the ‘wheat from the chaff’ and avoid the inclusion of irrelevant articles, you need a carefully configured exclusion strategy.

An easy way to do this is to formulate an exclusion criterion table to avoid duplicate, irrelevant and weak evidence. For example, you can assign a different exclusion code for:

• duplicate articles
• articles with inappropriate focus
• articles which is unrelated to subject matter
• weak evidence such as narrative reviews
• opinion articles
• animal studies

With these exclusion codes in mind, you can eliminate any articles that fall under your exclusion criteria. However, remember to keep a record of all the excluded articles .

4. Appraise properly

Once you identify the appropriate articles, you can start reviewing and appraising the clinical data.

Appraisal is the process of systematically reviewing research to determine its credibility, as well as its usefulness and applicability in the context. The clinical evidence should be evaluated thoroughly and objectively, with appropriate weighting given to the positive and negative aspects of each document and to each piece of research as a whole.

The best way to perform an effective appraisal is to set out an appraisal framework to determine the methodological quality, scientific validity, relevance to the Clinical Evaluation and weighting to the overall evaluation of each data source. It may be beneficial to assign a code to each element of appraisal along with a score for quality in that domain.

Our literature review process uses a 7-domain appraisal structure covering all aspects of study design, quality and relevance.

5. List out safety & performance benchmarks as you go

The final tip for performing literature search & review is to identify and list out the safety and performance benchmarks in the clinical evidence which is relevant to the context.

For example, if you are performing a literature search & review on a medical device, then the performance and safety benchmarks directly related to the device and those related to its comparable alternatives should be recorded. Do this as you go along , rather than ‘reverse engineering’ benchmarks at the end of the appraisal, as this will save considerable time.

A specific tip is to copy-paste benchmarks into a separate text editor file for reference later in the review process. This will help you to summarise the clinical evidence identified and to establish the state of the art (SOTA) in your literature review.

Do you need help with your literature search protocol & SOTA review?

Conducting a literature search & review is a vital and challenging aspect of clinical evaluation and other parts of the medical device regulatory process. Our team of professional medical writers and clinical experts are on hand to help, so please feel free to reach out for a no obligation discussion.

Or why not come along to one of our free ask-me-anything CER Clinics ? Get direct access to our medical writing team and ask them any CER or literature review related questions.

Sign up to our Regulatory Newsletter for the latest industry articles & news

Useful resources

• Literature Search & Review Services

Literature Search, SOTA Review process and Clinical Evaluation

• Full Clinical Evaluation Report Template

Related guide topics

• Literature Search Protocols, SOTA Reviews
• Clinical Evaluation Report (CER)
• Clinical Evaluation

Back to top

Related articles

Regulation (EU) 2024/1860 - Its impact on EU MDR and IVDR

How does the recent Regulation (EU) 2024/1860 amendment affect the EU MDR & IVDR?

MDCG 2024-10 - Orphan medical devices

How to apply MDR pre-market clinical evidence requirements to medical devices intended for limited usage.

A medical device's intended purpose - what is the point?

How do you define intended purpose, indication for use, intended clinical benefits, and claims?

EnableChat - Your AI-powered MDR and MDCG chatbot

Search the MDR and MDCG documents in seconds by asking EnableChat your questions.

Staying vigilant - A guide to searching for adverse events data

We discuss the pros and cons of existing adverse event databases for vigilance data searching.

What is Summary of Safety and Clinical Performance (SSCP)?

We explain what the SSCP is, when you'll need it and what its objectives are.

Medical Device 'Significant Changes' – Navigating EU MDR Article 120(3) using MDCG 2020-3 rev. 1

Understand what changes to your medical device are considered 'significant' under EU MDR (2017/745).

MDR or IVDR - A sibling rivalry?

A guide to easily understanding whether your device is a medical device or an in vitro diagnostic medical device (IVD).

What the latest Brexit U-turn means for CE Marking of medical devices in Great Britain

Will Great Britain continue to allow the use of the CE mark for medical devices beyond the 2024 deadline?

Gain confidence, reassurance and control over your EU MDR strategy

Find out how to build your own technical files within a guided framework while minimising financial outlays.

Still racing to achieve MDR compliance? A transition period update

On January 6th 2023, the EU commission has adopted the proposal to extend the transition rules of the EU MDR.

A medical device regulations guide for start-up companies

We present a 7-step guide to navigating regulatory requirements on a budget.

UKCA Marking of Medical Devices – An update on the status quo

We review recently updated requirements for UKCA marking and what it means for your regulatory strategy.

Choosing a CER writer for your MDR Clinical Evaluations

We've compiled a list of considerations that will help you make the right choice when choosing a CER writer.

How to achieve MDR Compliance for Class I medical devices

We outline a strategy for the regulatory compliance of Class I medical devices.

We help to demystify the process of systematic search & review of literature for Clinical Evaluation.

Five useful resources when writing a medical device CER

We outline five of the most useful and trustworthy Clinical Evaluation Report writing resources.

Five common pitfalls when writing a Clinical Evaluation Report

We illustrate five pitfalls when writing CERs and give you some tips to overcome them.

Five tips for making a medical device equivalence claim under the MDR

We'll show you what to keep in mind with regards to equivalance and Clinical Evaluation.

Keeping medical devices in market and maintaining CE-marks

The 4 golden rules to drive regulatory compliance with PMCF and vigilance data collection.

How PMCF goes beyond simple compliance

The wider benefits of a well-designed PMCF system include improving your products and your relationship with your clients.

Why you'll almost certainly need a PMCF system for your medical devices

We tell you what to be aware of under the EU MDR regarding PMCF and your medical devices.

The impact of Brexit on medical device regulatory compliance

How to ensure regulatory alignment of your devices in the territories affected by Brexit.

Is outside consulting support the answer to your MDR transition?

Getting ready for the MDR is a demanding process. Outsourcing might be your solution.

Increasing data entry compliance in PMCF studies

5 methods every medical device manufacturer should know to improve their Post-Market Clinical Follow-up studies.

Why medical doctors can drive MDR compliance

Working with the MDR requires knowing how to work with clinical evidence. Medical doctors are perfectly positioned to meet this requirement.

Software as a Medical Device

Unless you have spent time working with medical device legislation in the past, the idea that software could be a medical device may be rather unexpected.

Ensuring that clinical investigations work in practice

How can medical device manufacturers ensure valid clinical investigations when access to medical expertise remains limited?

Relaxing medical device regulatory requirements during a healthcare crisis

During the coronavirus pandemic, how far should we go when relaxing medical device regulatory requirements?

The new MDR compliance challenge

Across the industry, medical device companies are facing challenges in meeting the demands of the new Medical Device Regulations (MDR) 2017/745 framework.

Sources of Real World Evidence for MDR compliance

At Mantra Systems our objective is to make sure that our clients choose the method of real world data harvesting that is right for them.

More articles

Do you need help with your regulatory strategy?

Contact us today

10 September 2024: Due to technical disruption, we are experiencing some delays to publication. We are working to restore services and apologise for the inconvenience. For further updates please visit our website: https://www.cambridge.org/universitypress/about-us/news-and-blogs/cambridge-university-press-publishing-update-following-technical-disruption

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• > Journals
• > International Journal of Technology Assessment in Health Care
• > Volume 30 Issue 2
• > MEDICAL DEVICES EARLY ASSESSMENT METHODS: SYSTEMATIC...

Article contents

Medical devices early assessment methods: systematic literature review.

Published online by Cambridge University Press:  07 May 2014

• Supplementary materials

Objectives: The aim of this study was to get an overview of current theory and practice in early assessments of medical devices, and to identify aims and uses of early assessment methods used in practice.

Methods: A systematic literature review was conducted in September 2013, using computerized databases (PubMed, Science Direct, and Scopus), and references list search. Selected articles were categorized based on their type, objective, and main target audience. The methods used in the application studies were extracted and mapped throughout the early stages of development and for their particular aims.

Results: Of 1,961 articles identified, eighty-three studies passed the inclusion criteria, and thirty were included by searching reference lists. There were thirty-one theoretical papers, and eighty-two application papers included. Most studies investigated potential applications/possible improvement of medical devices, developed early assessment framework or included stakeholder perspective in early development stages. Among multiple qualitative and quantitative methods identified, only few were used more than once. The methods aim to inform strategic considerations (e.g., literature review), economic evaluation (e.g., cost-effectiveness analysis), and clinical effectiveness (e.g., clinical trials). Medical devices were often in the prototype product development stage, and the results were usually aimed at informing manufacturers.

Conclusions: This study showed converging aims yet widely diverging methods for early assessment during medical device development. For early assessment to become an integral part of activities in the development of medical devices, methods need to be clarified and standardized, and the aims and value of assessment itself must be demonstrated to the main stakeholders for assuring effective and efficient medical device development.

Access options

Markiewicz supplementary material.

This article has been cited by the following publications. This list is generated based on data provided by Crossref .

• Google Scholar

View all Google Scholar citations for this article.

Save article to Kindle

To save this article to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle .

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

• Volume 30, Issue 2
• Katarzyna Markiewicz (a1) , Janine A. van Til (a1) and Maarten J. IJzerman (a1)
• DOI: https://doi.org/10.1017/S0266462314000026

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox .

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive .

Reply to: Submit a response

- No HTML tags allowed - Web page URLs will display as text only - Lines and paragraphs break automatically - Attachments, images or tables are not permitted

Your details

Your email address will be used in order to notify you when your comment has been reviewed by the moderator and in case the author(s) of the article or the moderator need to contact you directly.

You have entered the maximum number of contributors

Conflicting interests.

Please list any fees and grants from, employment by, consultancy for, shared ownership in or any close relationship with, at any time over the preceding 36 months, any organisation whose interests may be affected by the publication of the response. Please also list any non-financial associations or interests (personal, professional, political, institutional, religious or other) that a reasonable reader would want to know about in relation to the submitted work. This pertains to all the authors of the piece, their spouses or partners.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Guidance for clinical evaluation under the medical device regulation through automated scoping searches.

Featured Application

1. introduction, 2. materials and methods, 2.2. mesh-based search, 2.3. bert-based search, 2.4. evaluation metrics, 2.5. workflow, 3.1. sme use cases, 3.2. clef 2018 ehealth tar, 4. discussion, 4.1. limitations, 4.2. outlook, 5. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, acknowledgments, conflicts of interest.

• Bianchini, E.; Francesconi, M.; Testa, M.; Tanase, M.; Gemignani, V. Unique device identification and traceability for medical software: A major challenge for manufacturers in an ever-evolving marketplace. J. Biomed. Inform. 2019 , 93 , 103150. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Minssen, T.; Gerke, S.; Aboy, M.; Price, N.; Cohen, G. Regulatory responses to medical machine learning. J. Law Biosci. 2020 , 7 , lsaa002. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Niemiec, E. Will the EU medical device regulation help to improve the safety and performance of medical ai devices? Digit. Health 2022 , 8 , 20552076221089079. [ Google Scholar ] [ CrossRef ]
• Malvehy, J.; Ginsberg, R.; Sampietro-Colom, L.; Ficapal, J.; Combalia, M.; Svedenhag, P. New regulation of medical devices in the EU: Impact in dermatology. J. Eur. Acad. Dermatol. Venereol. 2022 , 36 , 360–364. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• MedTech Europe. MedTech Europe Survey Report Analysing the Availability of Medical Devices in 2022 in Connection to the Medical Device Regulation (MDR) Implementation. Available online: https://www.medtecheurope.org/wp-content/uploads/2022/07/medtech-europe-survey-report-analysing-the-availability-of-medical-devices-in-2022-in-connection-to-the-medical-device-regulation-mdr-implementation.pdf (accessed on 14 June 2023).
• Makeenko, M.; Chevallier, T. Post-market Clinical Follow-up (PMCF) GAP Analysis for Legacy Devices Class III between the Medical Device Directive (MDD 93/42/EEC) and the Medical Device Regulation (MDR 2017/745). In BIOSTEC 2022 ; SciTePress: Setúbal, Portugal, 2022; pp. 273–280. [ Google Scholar ] [ CrossRef ]
• Beckers, R.; Kwade, Z.; Zanca, F. The EU medical device regulation: Implications for artificial intelligence-based medical device software in medical physics. Phys. Med. PM 2021 , 83 , 1–8. [ Google Scholar ] [ CrossRef ]
• Martelli, N.; Eskenazy, D.; Déan, C.; Pineau, J.; Prognon, P.; Chatellier, G.; Sapoval, M.; Pellerin, O. New european regulation for medical devices: What is changing? Cardiovasc. Interv. Radiol. 2019 , 42 , 1272–1278. [ Google Scholar ] [ CrossRef ]
• Vincent, P. Intra-articular hyaluronic acid in knee osteoarthritis: Clinical data for a product family (arthrum). with comparative meta-analyses. Curr. Ther. Res. Clin. Exp. 2021 , 95 , 100637. [ Google Scholar ] [ CrossRef ]
• Kanoulas, E.; Li, D.; Azzopardi, L.; Spijker, R. CLEF 2018 Technologically Assisted Reviews in Empirical Medicine Overview. CEUR Workshop Proc. 2018 , 2125 , 34. [ Google Scholar ]
• Cormack, G.; Grossman, M. Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review. arXiv 2015 , arXiv:1504.06868. [ Google Scholar ]
• Scells, H.; Zuccon, G.; Koopman, B. A comparison of automatic Boolean query formulation for systematic reviews. Inf. Retr. J. 2021 , 24 , 3–28. [ Google Scholar ] [ CrossRef ]
• Wang, S.; Scells, H.; Koopman, B.; Zuccon, G. Automated MeSH term suggestion for effective query formulation in systematic reviews literature search. Intell. Syst. Appl. 2022 , 16 , 200141. [ Google Scholar ] [ CrossRef ]
• Wang, S.; Li, H.; Scells, H.; Locke, D.; Zuccon, G. MeSH Term Suggestion for Systematic Review Literature Search. In Australasian Document Computing Symposium ; Association for Computing Machinery: New York, NY, USA, 2021; pp. 1–8. [ Google Scholar ] [ CrossRef ]
• Guo, J.; Cai, Y.; Fan, Y.; Sun, F.; Zhang, R.; Cheng, X. Semantic Models for the First-Stage Retrieval: A Comprehensive Review. ACM Trans. Inf. Syst. 2022 , 40 , 1–42. [ Google Scholar ] [ CrossRef ]
• Walsh, H.S.; Andrade, S.R. Semantic Search With Sentence-BERT for Design Information Retrieval. In Volume 2: 42nd Computers and Information in Engineering Conference (CIE) ; American Society of Mechanical Engineers: New York, NY, USA, 2022. [ Google Scholar ] [ CrossRef ]
• Achsas, S.; Nfaoui, E.H. Academic Aggregated Search Approach Based on BERT Language Model. In Proceedings of the 2022 2nd International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET), Meknes, Morocco, 3–4 March 2022; pp. 1–9. [ Google Scholar ] [ CrossRef ]
• Yang, E.; MacAvaney, S.; Lewis, D.D.; Frieder, O. Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review. In Advances in Information Retrieval ; Hagen, M., Verberne, S., Macdonald, C., Seifert, C., Balog, K., Nørvåg, K., Setty, V., Eds.; Springer International Publishing: Cham, Switzerland, 2022; pp. 502–517. [ Google Scholar ] [ CrossRef ]
• Rogers, A.; Kovaleva, O.; Rumshisky, A. A Primer in BERTology: What We Know About How BERT Works. Trans. Assoc. Comput. Linguist. 2020 , 8 , 842–866. [ Google Scholar ] [ CrossRef ]
• Zhan, J.; Mao, J.; Liu, Y.; Zhang, M.; Ma, S. An Analysis of BERT in Document Ranking. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual, 25–30 July 2020; ACM: New York, NY, USA, 2020; pp. 1941–1944. [ Google Scholar ] [ CrossRef ]
• Cooper, C.; Dawson, S.; Lefebvre, C. Searching for medical devices—Practical guidance. Res. Synth. Methods 2022 , 13 , 144–154. [ Google Scholar ] [ CrossRef ]
• Migliore, A. Technology assessment of innovative medical devices in europe. Expert Rev. Med. Devices 2016 , 13 , 217–219. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Hunter, K.E.; Webster, A.C.; Page, M.J.; Willson, M.; McDonald, S.; Berber, S.; Skeers, P.; Tan-Koay, A.G.; Parkhill, A.; Seidler, A.L. Searching clinical trials registers: Guide for systematic reviewers. BMJ 2022 , 377 , e068791. [ Google Scholar ] [ CrossRef ]
• Westrick, J.C.; Buchholz, S.W. Medical librarians’ knowledge and practices in locating clinical trials for systematic reviews. J. Med. Libr. Assoc. 2021 , 109 , 295–300. [ Google Scholar ] [ CrossRef ]
• Kiefer, A.-K.; Espig, K.; Moeller, K. A systematic review of recorded technical settings used in bipolar and monopolar transurethral resection of bladder tumors. Curr. Dir. Biomed. Eng. 2022 , 8 , 474–477. [ Google Scholar ] [ CrossRef ]
• Shatrov, K.; Blankart, C.R. After the four-year transition period: Is the European Union’s medical device regulation of 2017 likely to achieve its main goals? Health Policy 2022 , 126 , 1233–1240. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Survey on the EU Medical Devices Regulation by DIHK, SPECTARIS and MedicalMountains: First Assessment of the German Manufacturers of Medical Devices after the EU Medical Device Regulation (MDR) Came Into Force ; Deutscher Industrie-und Handelskammertag e.V.: Berlin, Germany; MedicalMountains GmbH: Tuttlingen, Germany; SPECTARIS. Deutscher Industrieverband für Optik, Photonik, Analysen-und Medizintechnik e.V.: Berlin, Germany, 2022; Available online: https://www.spectaris.de/fileadmin/Content/Medizintechnik/Startseite/Survey_on_the_EU_Medical_Devices_Regulation.pdf (accessed on 14 June 2023).
• Agyei, E.E.Y.F.; Pohjolainen, S.; Oinas-Kukkonen, H. Impact of Medical Device Regulation on Developing Health Behavior Change Support Systems. In Persuasive Technology ; Baghaei, N., Vassileva, J., Ali, R., Oyibo, K., Eds.; Springer International Publishing: Cham, Switzerland, 2022; pp. 1–15. [ Google Scholar ] [ CrossRef ]
• MacFarlane, A.; Russell-Rose, T.; Shokraneh, F. Search strategy formulation for systematic reviews: Issues, challenges and opportunities. Intell. Syst. Appl. 2022 , 15 , 200091. [ Google Scholar ] [ CrossRef ]
• Fujishiro, N.; Otaki, Y.; Kawachi, S. Accuracy of the Sentence-BERT Semantic Search System for a Japanese Database of Closed Medical Malpractice Claims. Appl. Sci. 2023 , 13 , 4051. [ Google Scholar ] [ CrossRef ]
• Tian, X.; Wang, J. Retrieval of Scientific Documents Based on HFS and BERT. IEEE Access 2021 , 9 , 8708–8717. [ Google Scholar ] [ CrossRef ]
• Health Technology and Cosmetics. Guidelines on Medical Devices: Clinical Evaluation: A Guide For Manufacturers and Notified Bodies Under Directives 93/42/EEC and 90/385/EEC. Available online: https://ec.europa.eu/docsroom/documents/17522/attachments/1/translations/en/renditions/native (accessed on 14 June 2023).
• European Commission. Guidelines on Medical Devices Meddev 2.7/1. Available online: https://www.google.com/search?client=firefox-b-d&q=meddev+2.7%2F1 (accessed on 13 March 2023).
• Hughey, J.; Schoenbachler, J. PMDB: A Relational Database for PubMed ; Zenodo: Geneva, Switzerland, 2022. [ Google Scholar ] [ CrossRef ]
• Schoenbachler, J.L.; Hughey, J.J. Pmparser and pmdb: Resources for large-scale, open studies of the biomedical literature. PeerJ 2021 , 9 , e11071. [ Google Scholar ] [ CrossRef ]
• Clinical Trials Transformation Initiative. Improving Public Access to Aggregate Content of ClinicalTrials.gov. Available online: https://aact.ctti-clinicaltrials.org/ (accessed on 27 January 2023).
• Lowe, H.J.; Barnett, G.O. Understanding and using the medical subject headings (mesh) vocabulary to perform literature searches. JAMA 1994 , 271 , 1103–1108. [ Google Scholar ] [ CrossRef ]
• Mork, J.G.; Jimeno-Yepes, A.; Aronson, A.R. The NLM Medical Text Indexer System for Indexing Biomedical Literature. In Proceedings of the First Workshop on Bio-Medical Semantic Indexing and Question Answering, a Post-Conference Workshop of Conference and Labs of the Evaluation Forum 2013 (CLEF 2013), Valencia, Spain, 27 September 2013; Available online: https://ceur-ws.org/Vol-1094/bioasq2013_submission_3.pdf (accessed on 14 June 2023).
• Robertson, S.E.; Jones, K.S. Relevance weighting of search terms. J. Am. Soc. Inf. Sci. 1976 , 27 , 129–146. [ Google Scholar ] [ CrossRef ]
• Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv 2018 , arXiv:1810.04805. [ Google Scholar ]
• Deka, P.R.I.T.A.M.; Jurek-Loughrey, A.N.N.A.; Deepak, P. Improved Methods to Aid Unsupervised Evidence-Based Fact Checking for Online Heath News. J. Data Intell. 2022 , 3 , 474–504. [ Google Scholar ] [ CrossRef ]
• Gu, Y.; Tinn, R.; Cheng, H.; Lucas, M.; Usuyama, N.; Liu, X.; Naumann, T.; Gao, J.; Poon, H. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. ACM Trans. Comput. Healthc. 2022 , 3 , 1–23. [ Google Scholar ] [ CrossRef ]
• Liu, T.-Y. Learning to Rank for Information Retrieval. Found. Trends® Inf. Retr. 2007 , 3 , 225–331. [ Google Scholar ] [ CrossRef ]
• Cormack, G.; Grossman, M. Technology-Assisted Review in Empirical Medicine: Waterloo Participation in CLEF eHealth 2018. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF), Avignon, France, 10–14 September 2018. [ Google Scholar ]
• Burgard, T.; Bittermann, A. Reducing Literature Screening Workload With Machine Learning. Z. Für Psychol. 2023 , 231 , 3–15. [ Google Scholar ] [ CrossRef ]
• Yamada, T.; Yoneoka, D.; Hiraike, Y.; Hino, K.; Toyoshiba, H.; Shishido, A.; Noma, H.; Shojima, N.; Yamauchi, T. Deep neural network for reducing the screening workload in systematic reviews for clinical guidelines: Algorithm validation study. J. Med. Internet Res. 2020 , 22 , e22422. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Huang, W.; Sun, S.; Lin, X.; Li, P.; Zhu, L.; Wang, J.; Chen, C.L.P.; Sheng, B. Unsupervised fusion feature matching for data bias in uncertainty active learning. IEEE Trans. Neural Netw. Learn. Syst. 2022; early access . [ Google Scholar ] [ CrossRef ]
• Wallace, B.C.; Small, K.; Brodley, C.E.; Trikalinos, T.A. Active learning for biomedical citation screening. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, 25–28 July 2010; ACM: New York, NY, USA, 2010; pp. 173–182. [ Google Scholar ] [ CrossRef ]
• Cohen, D.; Jordan, S.M.; Croft, W.B. Learning a Better Negative Sampling Policy with Deep Neural Networks for Search. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, Santa Clara, CA, USA, 2–5 October 2019; ACM: New York, NY, USA, 2019; pp. 19–26. [ Google Scholar ] [ CrossRef ]
• Eto, M. Extended co-citation search: Graph-based document retrieval on a co-citation network containing citation context information. Inf. Process. Manag. 2019 , 56 , 102046. [ Google Scholar ] [ CrossRef ]
• Wang, X.; Macdonald, C.; Tonellotto, N.; Ounis, I. Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, Virtual, 11 July 2021; ACM: New York, NY, USA, 2021; pp. 297–306. [ Google Scholar ] [ CrossRef ]
• Carvallo, A.; Parra, D.; Lobel, H.; Soto, A. Automatic document screening of medical literature using word and text embeddings in an active learning setting. Scientometrics 2020 , 125 , 3047–3084. [ Google Scholar ] [ CrossRef ]
• Ein-Dor, L.; Halfon, A.; Gera, A.; Shnarch, E.; Dankin, L.; Choshen, L.; Danilevsky, M.; Aharonov, R.; Katz, Y.; Slonim, N. Active Learning for BERT: An Empirical Study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, 16–20 November 2020; Association for Computational Linguistics: Stroudsburg, PA, USA, 2020; pp. 7949–7962. [ Google Scholar ] [ CrossRef ]
• Salvador-Oliván, J.A.; Marco-Cuenca, G.; Arquero-Avilés, R. Errors in search strategies used in systematic reviews and their effects on information retrieval. J. Med. Libr. Assoc. 2019 , 107 , 210–221. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Tasneem, A.; Aberle, L.; Ananth, H.; Chakraborty, S.; Chiswell, K.; McCourt, B.J.; Pietrobon, R. The database for aggregate analysis of clinicaltrials.Gov (aact) and subsequent regrouping by clinical specialty. PLoS ONE 2012 , 7 , e33677. [ Google Scholar ] [ CrossRef ]
• Bethesda (MD): National Library of Medicine. Statistical Reports on MEDLINE®/PubMed® Baseline Data. Available online: https://www.nlm.nih.gov/bsd/licensee/2022_stats/2022_LO.html (accessed on 14 June 2023).
• Tullu, M.S. Writing the title and abstract for a research paper: Being concise, precise, and meticulous is the key. Saudi J. Anaesth. 2019 , 13 (Suppl. 1), S12–S17. [ Google Scholar ] [ CrossRef ]
• Klingmann, I.; Paul, M.; Bereczky, T.; Plate, A.; Breithaupt-Grögler, K.; Schindler, T.M.; Geissler, J. Veränderungen durch die neue EU-Verordnung 536/2014 zu klinischen Prüfungen aus Patientensicht. Bundesgesundheitsblatt Gesundh. Gesundh. 2023 , 66 , 68–74. [ Google Scholar ] [ CrossRef ]
• Ebeid, I.A. Medgraph: A semantic biomedical information retrieval framework using knowledge graph embedding for pubmed. Front. Big Data 2022 , 5 , 965619. [ Google Scholar ] [ CrossRef ]
• Zarrinkalam, F.; Kahani, M. A multi-criteria hybrid citation recommendation system based on linked data. In Proceedings of the 2012 2nd International eConference on Computer and Knowledge Engineering (ICCKE), Mashhad, Iran, 18–19 October 2012; pp. 283–288. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Burri, K.G. Intelligent use of artificial intelligence for systematic reviews of medical devices. Med. Writ. 2019 , 28 , 22–27. [ Google Scholar ]

Click here to enlarge figure

Share and Cite

Tang, F.-S.K.-B.; Bukowski, M.; Schmitz-Rode, T.; Farkas, R. Guidance for Clinical Evaluation under the Medical Device Regulation through Automated Scoping Searches. Appl. Sci. 2023 , 13 , 7639. https://doi.org/10.3390/app13137639

Tang F-SK-B, Bukowski M, Schmitz-Rode T, Farkas R. Guidance for Clinical Evaluation under the Medical Device Regulation through Automated Scoping Searches. Applied Sciences . 2023; 13(13):7639. https://doi.org/10.3390/app13137639

Tang, Fu-Sung Kim-Benjamin, Mark Bukowski, Thomas Schmitz-Rode, and Robert Farkas. 2023. "Guidance for Clinical Evaluation under the Medical Device Regulation through Automated Scoping Searches" Applied Sciences 13, no. 13: 7639. https://doi.org/10.3390/app13137639

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

SYSTEMATIC REVIEW article

A systematic review of medical equipment reliability assessment in improving the quality of healthcare services.

• 1 Department of Biomedical Engineering, Faculty of Engineering, Universiti Malaya, Kuala Lumpur, Malaysia
• 2 Engineering Services Department, Ministry of Health Malaysia, Putrajaya, Malaysia
• 3 Department of Electrical and Electronic Engineering, Faculty of Engineering and Built Environment, Universiti Sains Islam Malaysia, Nilai, Malaysia
• 4 School of Computer Engineering, Kalinga Institute of Industrial Technology, Deemed to Be University, Bhubaneswar, India

Medical equipment highly contributes to the effectiveness of healthcare services quality. Generally, healthcare institutions experience malfunctioning and unavailability of medical equipment that affects the healthcare services delivery to the public. The problems are frequently due to a deficiency in managing and maintaining the medical equipment condition by the responsible party. The assessment of the medical equipment condition is an important activity during the maintenance and management of the equipment life cycle to increase availability, performance, and safety. The study aimed to perform a systematic review in extracting and categorising the input parameters applied in assessing the medical equipment condition. A systematic searching was undertaken in several databases, including Web of Science, Scopus, PubMed, Science Direct, IEEE Xplore, Emerald, Springer, Medline, and Dimensions, from 2000 to 2020. The searching processes were conducted in January 2020. A total of 16 articles were included in this study by adopting Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA). The review managed to classify eight categories of medical equipment reliability attributes, namely equipment features, function, maintenance requirement, performance, risk and safety, availability and readiness, utilisation, and cost. Applying the eight attributes extracted from computerised asset maintenance management system will assist the clinical engineers in assessing the reliability of medical equipment utilised in healthcare institution. The reliability assessment done in these eight attributes will aid clinical engineers in executing a strategic maintenance action, which can increase the equipment's availability, upkeep the performance, optimise the resources, and eventually contributes in providing effective healthcare service to the community. Finally, the recommendations for future works are presented at the end of this study.

Introduction

The growing sophistication of medical equipment has significantly improved the individual and society's health ( 1 ). The advancement has improved survivability in the face of disease or injury and greatly enhanced patients' life quality through an improved diagnosis and therapeutic results. Managing assets and facilities is one of the significant features in ensuring the continuity of primary and support business activities in healthcare services ( 2 ). The delivery of healthcare services to the communities are significantly affected without effective management implementation ( 3 – 5 ). Medical equipment is a crucial asset that substantially contributes to the effectiveness and healthcare services quality enhancement ( 6 , 7 ). As the medical equipment aids various services in the healthcare sector, the management representative, such as clinical engineers, must monitor and upkeep the assets by performing several maintenances works throughout the equipment life cycle ( 8 , 9 ).

Maintenance management of medical equipment is crucial to ensure that a machine operates in accordance with manufacturer specifications and guarantees the patients and users safety ( 10 ). Failure of medical equipment may affect the healthcare services effectiveness and cause severe injury to the patients and harm the environment ( 11 ). Bahreini et al. ( 12 ) summarised that the affecting factors are management, resources, information bank, service, inspection, education, and quality control. Performance assessment is one of the activities that can be carried out regularly throughout the maintenance and repair phase to determine the medical equipment's actual condition.

Executing the assessment requires information concerning medical equipment features to produce the expected output. The expected output will assist healthcare management or clinical engineers in making essential decisions on maintenance management practises to enhance the reliability and availability of medical equipment. Furthermore, specific studies on assessment techniques within the South East Asia region, particularly in compliance with the Malaysian standard for managing medical equipment maintenance, are still lacking. In developing the present systematic review, the following research questions were addressed:

• What are the significant parameters required on the medical equipment to be applied for the reliability assessment from the previous studies?

• How do these parameters applicable to the Malaysian standard practises for managing the maintenance of medical equipment?

Selecting the significant parameters to be considered for medical equipment reliability assessment is very crucial in ensuring optimum healthcare services. In this study, the identification of these significant parameters can be applied for various types of medical equipment utilised in any healthcare institutions. In addition, we provide the review on feasibility of prediction of medical equipment reliability analysis using artificial intelligence (AI) and/or machine learning (ML) techniques based on these parameters throughout the maintenance phase of the medical equipment's life cycle. This study also leads to the revealing of the gap and novelty. The identified parameters will contribute to the comprehensive and strategic maintenance management of medical equipment, which cover three main elements of preventive maintenance (PM), corrective maintenance (CM), and replacement plan (RP). Furthermore, the reliability assessment using these parameters may fulfil and improve the medical equipment maintenance's national standard. Based on the study undertaken, none of included studies contributed on these three aspects and correlates the parameters with the relevant standards. Hence, the study aimed to identify the significant parameters of medical equipment by undertaking a systematic review of previous studies and correlate with the Malaysian standard of medical equipment maintenance management.

Materials and Methods

Literature search.

The systematic literature review was performed by applying the published standard, namely PRISMA in evaluating and rigorously analysing the articles related to medical equipment assessment in the databases ( 13 ). Besides, the inclusion and exclusion processes of the relevant current studies were thoroughly performed. The examination of the included study is coded to achieve the systematic review's objective in the subject area.

The studies related to medical equipment assessment was from two primary databases, namely Web of Science and Scopus. The databases cover more than 256 studies field, including engineering and computer science studies that may increase the comprehensiveness and qualities of the article ( 14 , 15 ). According to Younger ( 16 ), several established databases should be included to enhance the possibility of achieving the relevant articles in the subject area. In this study, the selected additional databases were PubMed, Science Direct, IEEE Xplore, Emerald, Springer, Medline, and Dimensions.

Article Selection

This stage elaborates the articles selection process. There are three steps in selecting the relevant articles, namely identification, screening, and eligibility.

Identification

The identification and selection of the relevant studies comprise four main stages. Firstly, the subject areas' keywords were identified. The thesaurus, encyclopaedia and past researches were referred to construct appropriate keywords. Secondly, search string algorithms were developed from the keywords in January 2020 based on the Web of Science and Scopus databases' characteristics, as illustrated in Table 1 . Next, several inclusions and exclusion criteria were determined to retrieve the articles from both databases (Refer to Table 2 ). These criteria were set because only the latest research articles in the subject area were retrieved to minimise the possibility of irrelevant topic inclusion. Besides, only English articles were considered for easier analysis preparation. Subsequently, these two search strings were applied in the databases' advanced search. Resultantly, 183 articles from Web of Science and 505 articles from Scopus were retrieved. Similar keywords were applied in the seven other databases, where 64 articles were identified. Moreover, identification of relevant studies was carried via other methods, which are websites, organisations, and citation searching ( 17 ). By using the similar keywords, there were 98 references were identified, Thus, 852 references consist of the articles and reports were retrieved in the identification stage.

Table 1 . The search strings for Web of Science and Scopus databases.

Table 2 . The inclusion and exclusion criteria.

The 852 articles and reports were divided into two to remove duplication and exclude non-related subject areas or topics during the screening process. There were 38 and 19 repeated articles in the databases and other methods, respectively. Therefore, these duplicated articles were removed, and the remaining 716 articles and 79 reports progressed to a further screening process. Three features were properly examined during the screening process: the title, keywords, and abstract. Furthermore, several considerations were considered while examining the three features. Firstly, the general terms of medical equipment or medical device or other specific equipment categorised under these general terms were mentioned in the title and keywords. Secondly, an indication of the quantitative method in assessing the medical equipment performance was depicted in the abstract. Consequently, only 85 articles and 21 reports were selected to progress to the following step.

Eligibility

This step involved reviewing the articles' full text to ensure that the 85 research articles and 21 reports were eligible to be synthesised and analysed. The articles' significant contents were comprehensively scrutinised to confirm that the inclusion and exclusion criteria were fulfilled. Essential elements such as study aim, input parameters, methodology technique, expected output, and desired outcomes were thoroughly assessed. Subsequently, 69 articles and 21 reports were excluded due to not utilising the quantitative method to assess the medical equipment performance and not empirical studies. Besides, an additional two relevant articles were included based on hand-searching. Therefore, a total of 16 remaining articles were included in this study, as illustrated in Figure 1 .

Figure 1 . PRISMA flow chart of the study adapted from Page et al. ( 17 ).

Quality Assessment and Data Extraction

The qualitative analysis technique was used to assess the remaining articles. The first, second, and sixth authors performed the quality assessment of selected articles. The articles were categorised into high, moderate, and low levels that reflect the aim, input parameters, methodology technique, expected output, and desired outcomes ( 18 ). The articles must reach a high level and be agreed upon by all the authors. The compilation of extracted information was carried out by the first, second, and sixth authors and synthesised in an organised table. The third, fourth, and fifth authors subsequently checked all the synthesised data. The synthesised data was categorised by applying the thematic analysis. According to the Active Medical Device Maintenance Management developed by the Department of Standard, Malaysia, the established categories were correlated with the features. The result of input features categorisation was prudently discussed among authors. Any discrepancies or inconsistencies were resolved by consensus and until reaching reviewers agreement.

Overall Background and Studies Findings

The analysis was carried out on 16 articles included in this study, as presented in Table 3 . Based on this table, we concluded that none of the studies performed comprehensive analyses which include PM, CM, and RP. The selected studies included either one of the medical equipment reliability assessments of PM, CM, RP, and/or combination of either type of assessment. All articles were reviewed, and the motivations of each study were analysed and extracted. The articles' common traits were identified, and potential research gaps were determined. Currently, no proper protocol and early intervention exercise in assessing the performance of medical equipment were reported. The healthcare organisation faces difficulty in implementing effective maintenance management of medical equipment without proper methodological procedure and planning. Thus, the medical equipment is unable to correctly operate and could be harmful to patients and users.

Table 3 . Authors, maintenance activity, output indicator, and outcomes.

Since a large number of medical equipment and multiple functions are utilised in healthcare institutions, the equipment shall be monitored and correctly maintained to sustain performance and safety levels. However, maintenance management could be challenging if the healthcare provider encounters several problems regarding insufficient competent personnel and available resources, such as replacement parts and funds. According to the World Health Organisation (WHO), initial expenses, and operating expenditures are two categories of required financial resources in maintaining medical equipment ( 35 ). Corciova et al. ( 36 ) added that maintenance expenses represent a significant portion of the entire healthcare system which required 15–60% of the total cost to operate. Improper maintenance may affect performance and safety which greatly gave significant impact on the expenditure of healthcare institutions ( 37 ). Wu et al. ( 38 ) proved that practising effective maintenance management within 2 years improves the medical equipment availability and minimises the operating costs which exceeded one million dollars.

The computerised inventory system significantly assists healthcare management in managing equipment and maintenance activities. Applying the appropriate methodological technique in processing big data generates useful indicators that may assist clinical engineers in strategising the maintenance planning and further action course. The identification of the medical equipment criteria is essential to produce valuable indicators. Based on the analysis from the included articles, the justification for the criteria identification was referred from the previous literature, data collection and extraction, expert judgement via survey, input based on customer requirement, and adapting from international standards and national guidelines.

The relevant data were collected, processed, calculated, and analysed accordingly based on the identified criteria. Only one scientific methodological technique was involved in generating the expected output by referring to the 12 articles ( 19 – 30 ). Nevertheless, according to Ben Houria et al. ( 31 ), a combination of three techniques generated the expected output. The combination of two techniques was observed in the studies performed by Oshiyama et al. ( 32 ), Saleh and Balestra ( 33 ), and Ismail et al. ( 34 ), respectively. The proposed techniques were tested on the real dataset of various types of medical equipment particulars and maintenance information within a specific period.

The conclusion from the review of the 16 articles is that the healthcare institutions are capable of optimising the maintenance cost, improving the monitoring activity, managing the maintenance activities with available workforces and resources, prioritising PM and CM, assessing the equipment's actual lifespan for the purpose RP, selecting the best maintenance management strategy based on the current situation by referring to the output indicator.

Main Findings

The outcomes from the selected articles using thematic analysis produced eight categories of significant parameters in assessing the medical equipment condition and reliability. The application of thematic analysis was carried out to develop the appropriate themes of medical equipment's parameters. Based on the thematic analysis, these parameters are found to be significant for the AI/ML network as the input parameters. The initial stage of theme development procedures was the compilation of each parameter extracted from the 16 selected articles. In this phase, the categories of input parameters were properly analysed to extract the description used in the selected articles to address the research gaps. Then, in the second phase, various terms of input parameters were converted into general category via the themes, applications, or ideas classification. From the analysis, we found out that there were many terms used in previous studies, however, several of them can be addressed under the same group. Eventually, the thematic analysis has generated a total of eight categories of input parameters, which described the characteristics of medical equipment utilised in healthcare institutions.

The observation was also carried out according to the studies outcomes. There were seven categories of input parameters extracted from the selected articles as tabulated in Table 4 . In general, the outcome was to classify the medical equipment in accordance with maintenance management activities. The medical equipment maintenance activities comprise of PM, CM, and RP. The prioritisation was made based on the medical equipment characteristics toward the strategic maintenance management activity.

Table 4 . Categories of input parameters.

The generated eight categories of input parameters will assist the clinical engineers to perform the medical equipment reliability assessment. Understanding of the equipment reliability may lead to proper maintenance activity, which subsequently increase the availability of medical equipment with optimised resources. The measurement of these parameters using AI/ML techniques will comprehensively enhance the monitoring of medical equipment performance and utilisation status through predictive maintenance model. This predictive model is able to mitigate the potential failures, deterioration, and obsolescence. Findings from the selected articles produced eight categories of the significant input parameters in assessing the medical equipment condition, as tabulated in Table 4 .

Equipment Features

Equipment features comprise several characteristics designed for the equipment and exist since the unit is manufactured. One of the parameters used to assess the medical equipment condition is age ( 19 , 20 , 22 – 24 , 26 , 27 , 29 – 31 ). The equipment age reflects the overall condition of the equipment. This is because, the equipment is typically performed well at an early age, and fewer failures are observed. However, as the age increases, the equipment starts to degrade. In another studies conducted by Badnjevic et al. ( 20 ) and Kovacevic et al. ( 19 ), manufacturer's name and equipment's modality are another two parameters that were included in their reliability assessment. Prior to that, Faisal et al. ( 24 ), included the service availability as one of the input parameters in assessing the equipment condition for the RP prioritisation. The service availability of the medical equipment includes warranty, documentation, training, and compatible spare part. Meanwhile, studies by Saleh et al. ( 21 ) and Saleh and Balestra ( 33 ) also considered equipment complexity as the input parameter in their medical device reliability assessment.

The function of the equipment here reflects on the intended use of the equipment in healthcare services delivery. The equipment function is divided into several forms of services namely life support, therapeutic, diagnostic, analytical, and miscellaneous.

Maintenance Requirement

Maintenance requirements involve activities and tasks to ensure that the medical equipment is sustained in the expected condition in terms of functionality and physical. The complexity of carrying out the maintenance procedure is different among the equipment types. The maintenance, servicing, or restoration of this equipment requires a skilled person to dismantle and replace the replacement or faulty part. The procedure is also time-consuming. Thus, the equipment can be unavailable for an extended period if the performance of maintenance activity is carried out ineffectively.

Performance

The reliability of the medical equipment highly depended on the performance that can be measured from the efficiency and uptime. The equipment effectiveness can be observed from the usage and service life. The performance of medical equipment should be vitally monitored and ensured as described by the manufacturer. The excellent performance of medical equipment can mitigate the interruption of healthcare services to the public.

Risk and Safety

In delivering the healthcare services such as diagnosis and treatment, patients and clinicians must be kept safe without exposure to any hazard that may cause severe injury. The risk and safety of the equipment can be predicted by studying the failure aspects ( 23 , 28 , 30 ). The authority may issue the recalls and hazards alert if any incident occurs involving medical equipment utilisation by instructing the user to immediately stop using the equipment to prevent further possible danger to clinicians and patients ( 23 , 30 ). In addition, hazards may when mishandling, misdiagnosis, inappropriate treatment or error are made by the operator ( 30 , 31 ). Hazards may also exist from the operation or physical of the medical equipment ( 21 , 25 , 33 ).

Availability and Readiness

The availability and readiness of medical equipment are vital to ensure that healthcare services to patients can be delivered without compromise. The category consists of the alternatives and backup units, device criticality, and user acceptability. The breakdown of equipment is inevitable due to normal wear and tear, or ageing may interfere with the effectiveness of healthcare services. Moreover, the importance of healthcare services will cause equipment to become a critical necessity due to unavoidable circumstances ( 25 – 27 , 29 – 31 ). Thus, alternative units must be ready for critical times ( 24 , 25 , 27 – 31 ).

The assessment of the medical equipment condition can also be undertaken by observing the unit utilisation level. The utilisation level can be affected by the location of equipment and the situation where the equipment is used to deliver healthcare services to the public ( 21 , 22 , 26 , 33 ). According to the authors, the frequency of medical equipment usage and turning to be essential depending on the type and activity of healthcare services provided to patients, such as anaesthetising, operating theatres, and others. Furthermore, the equipment may be extensively or rarely used depending on the healthcare services situation.

Cost is one of the crucial aspects in managing medical equipment maintenance and replacement activities. According to the studies performed by Ben Houria et al., reducing the cost of operations involving medical equipment is crucial. The operational costs must be below the allocated budget ( 31 ). Findings from Faisal et al. ( 24 ) reported that maintenance costs should not be over 25% of the procured medical equipment cost over the past 3 years. In other study conducted by Oshiyama et al. ( 32 ) suggested that the CM cost should be within 3 to 5% of the medical equipment purchased price. According to Hutagalung et al. ( 29 ), maintenance costs can be reduced by enhancing the availability of medical equipment through effective maintenance management. With regards to the previous findings stated before, the maintenance cost highly reflected on the reliability of the medical equipment. The frequent failures do not only lead to the excessive maintenance tasks and disruption to the healthcare services, but also involve extra expenses. When the equipment requires high maintenance operation and the imposed cost reached to the specific limits, the equipment is no longer reliable and fit for utilisation. This condition is known as beyond economic repair.

Overall Findings

Previous studies demonstrated the importance of assessing medical equipment in planning for necessary action within healthcare institutions. The first important consideration in developing the medical equipment performance assessment is determining the appropriate input parameters ( 12 ). However, no single technique can be applied to all the input parameters. The selection of input parameters must be appropriate and applicable to the expected output. According to Mahfoud et al. ( 39 ), the outcome of the medical equipment assessment associates with the maintenance strategies. The availability of an existing dataset comprising medical equipment details and maintenance history is one of the factors in selecting the appropriate input parameters. The difference in input parameters applied can be processed to generate similar output.

The second consideration is the optimum processing technique based on the myriad of medical equipment data. As mentioned earlier, many scientific methods were developed which can be used to compute the input data and eventually generate an expected output for assessment purposes ( 40 ). However, the ML technique application is observed to be a better technique compared to the conventional techniques. This is due to the capability of the ML algorithm in testing the predictive high output accuracy by applying the accurate and significant input data.

The results obtained from the studies made by Badnjevic et al. ( 20 ) and Kovacevic et al. ( 19 ) showed that the generated output achieved above 89% accuracy where Random Forest and Decision Tree reached around 99% of accuracy in predicting both selected medical equipment performance. Therefore, both authors concluded that improved supervision, quality and safety in managing medical equipment maintenance could be achieved which eventually optimised the cost of maintenance. However, the ML techniques utilised in both studies were developed based on only one type of medical equipment. Consideration of applying to various types of medical equipment would be more practical to be utilised in healthcare facility management. This is because, various types of medical equipment have difference functionality and required specific assessment to ensure their reliability to be used in healthcare services.

The third consideration in developing the medical equipment condition assessment technique is to determine the expected output. One of the indications in identifying the expected output is observing the trend of the occurred problems ( 41 ). From the list of observations, the trend is translated into a specific objective to resolve the problem. The review concludes that the clinical engineers faced several common issues, such as the unavailability of medical equipment due to malfunctioning, insufficient workforces (i.e., competent technical staff), and limited resources (i.e., limited resources budget). Effective maintenance management must be established to overcome these problems and prevent severe consequences. The prioritisation by assessing the existing medical equipment condition can be undertaken while working within the current workforce and resources.

The Medical Equipment Reliability Assessment From the Malaysian Perspective

The Malaysian government spent approximately RM27 million in 2018 for new procurement and upgrading initiatives of medical equipment in public healthcare facilities to provide efficient healthcare services to the public ( 42 ). The Malaysian government also executed a new leasing programme involving six main medical equipment for 5 years starting from 2019, comprising a maintenance scheme with an approximate cost of RM19.7 million.

In the private sector, KPJ Healthcare Berhad, a leading private healthcare service in Malaysia with more than 20 hospitals throughout the country, procured medical equipment worth around RM136 million in 2019, showing an increment of 32% from the previous year ( 43 ). The evidence from both sectors indicates that massive investment in procurement and maintenance of medical equipment is necessary in delivering effective healthcare services to the community. Therefore, the efficient maintenance management of medical equipment during operations is vital to maximise the life span of the unit and ensure the investment is worthy.

During the eleventh Malaysia Plan, there was a need of highly technological medical equipment to meet the essentials for various kind of diseases advanced management ( 44 ). These critical machines such as computerised tomography (CT) and magnetic resonance imaging (MRI) were required in facilitating the medical practitioners for detecting, diagnosing, and treating the critical diseases. The National Medical Devices Survey was conducted in both public and private healthcare sectors and found out that the ratio of MRI number and population was two per million, whereas CT was 4 per million. This finding indicated the lower ratios than of Organisation for Economic Co-operation and Development (OECD) countries.

Referring to the Ministry of Health report for three consecutive years starting from 2017, the percentages of hospitals and public health facilities outpatient attendees had slightly increased in average of 2.64%, whereas the hospitals day care attendees had significantly escalated to 15.8% ( 45 – 47 ). Furthermore, the hospitals admissions had marginally increased to 5.96%. This indication drove the Malaysian government to expand the secondary and tertiary care services. From the year of 2015–2018, the number of hospital beds was increased by 3.3%, in which the increment of 11% applied to intensive care units ( 48 ). Although the improvement was made, the existing ratio of 1.9 beds to every 1,000 Malaysia populations is still below than the initial target. From the initiative programme implementation, the Malaysian government provides 67% of total beds in the country. Other expansion programmes that initiated by the Malaysian government were the development a new replacement hospital in east-coast region, the extension of the new complexes in the existing hospitals, and the new hospital in central region. These development activities require the expansion of facility, equipment, manpower, and services.

Apart from this expansion initiative, the medical-based day care services such as paediatric, oncology, and haematology require specific medical equipment for chemotherapy, blood transfusion, and haemodialysis ( 49 ). Therefore, the Ministry of Health Malaysia has come out with a general policy to enhance the availability of medical equipment for strengthen the healthcare services in the country. Besides, based on the inspection carried out by the internal audit of Ministry of Health Malaysia has recommended that the improvement of medical equipment is required to facilitate the healthcare services to the public ( 50 ).

The application of medical equipment in facilitating the healthcare services is crucial. The Malaysian government provided a huge amount of funds in procuring and maintaining the medical equipment in the country. The procurement of medical equipment includes for a new development of hospitals and replacement of the disposed units due to beyond economic repair ( 42 ).

Subsequently, this sub-section discusses the correlation on eight categories of input parameters as set by the Malaysian standard, namely the Code of Practise for Good Engineering Maintenance Management of Active Medical Devices (MS 2058:2018) ( 51 ). The primary function of this body is to encourage and promote global competitiveness through reliable standardisation and accreditation services governed by the Standards of Malaysia Act 1996 Act 549 ( 52 ). Several normative references comprising other interconnected MS 2058:2018 guidelines and national acts are available to develop a comprehensive standard and comply with the legislative requirement.

One crucial related act, specifically for managing the medical equipment, is the Medical Device Act 2012 ( 53 ). This act is intended to provide statutory regulation for medical devices in Malaysia and shall be complied with by all relevant parties, such as authorised representatives, manufacturers, and service providers. This act is also under the enforcement of the Malaysian governmental regulatory body, namely the Medical Device Authority, based on the establishment of the Medical Device Authority Act 2012 ( 54 ). Act 737 regulates the entire life cycle of medical equipment covering three main phases, namely pre-market, placing in market and post-market.

According to Annex P of MS 2058:2018, ten factors are proposed to assess the medical equipment for the RP. Based on the observation and comparison made with the included studies as shown in Table 5 , these factors can also be used as an input to assess the medical equipment condition for maintenance prioritisation.

Table 5 . Comparison between factors proposed in MS 2058:2018 and included studies.

Firstly, according to the observation of the factors proposed in the MS 2058:2018, the asset age is directly similar to the first category grouped based on the analysis in this study, namely the equipment features. Furthermore, most authors highly utilise equipment age to obtain an indication of prioritising maintenance and RP. The second factor that is similar to the equipment feature category was obsolescence. This factor is quite identical to service support because if there is no service support in the market, the restoration work involves replacement parts or any maintenance services for related equipment can be delivered. Therefore, the factors of asset age and obsolescence are similar to equipment features.

The results of comparisons between all the factors proposed in the MS 2058:2018 with two of the eight categories, namely function and maintenance requirement, found no similarity. Next, the performance category comprises several parameters involving efficiency, failure, downtime, uptime, and the number of corrective maintenances performed. In comparison with the MS 2058:2018, the performance category seems equivalent to the frequency of breakdown where the medical equipment performance can be measured by assessing the failure rate. Furthermore, factors such as asset status and asset condition seem to be related to this category.

The next category initiated in this study was risk and safety. This category is crucial to mitigate any potential hazards posed to the patient and clinician. From the comparison made, the factor of safety alert proposed in MS 2058:2018 correlated with the risk and safety category. The correlation was due to the risk and safety category involving the recalls and hazards alerts that can be declared or issued by the local authority body ( 53 ), manufacturer or locally authorised representative. The availability and readiness were in a category consisting of the element of correlation between equipment and service criticality. In MS 2058:2018, the factors that were observed as most similar were the availability of backup equipment and user recommendation. These factors demonstrate the criticality in ensuring equipment availability in assisting the healthcare services at the clinically acceptable level.

Asset usage is one of the 10 factors proposed in MS 2058:2018 that indicates the extensive utilisation level of the medical equipment. Direct similarity with the category of utilisation was apparent when compared against the categories in this study. A direct similarity can be identified between the maintenance cost factor proposed in MS 2058:2018 and the cost categorised in this study. Table 6 tabulates the correlation between the eight categories summarised based on the review in this study with the 10 factors proposed in MS 2058:2018.

Table 6 . Correlation between study categories and factors in MS2058:2018.

The national standard of MS 2058:2018 helps the clinical engineers significantly in managing the maintenance of medical equipment in Malaysia. There are several vital relevant acts and standard were referred for the development and compilation of this national standard. It covers various types of equipment in all maintenance stages starting from equipment acceptance to disposal process. This standard also includes main and sub-element of maintenance activities, which are PM, CM, and RP. The ultimate purpose is to ensure a proper maintenance is performed for continuous performance of medical equipment.

The adherence of MS 2058:2018 may optimise the performance of medical equipment utilised in healthcare institution. The optimised performance of medical equipment reduces error while delivering the healthcare services to public. Any difference in measurement could give bad impact on the healthcare services especially to the patients. It is important to ensure that the equipment can give accurate and precise measurement during therapeutic, diagnostic, and analytic procedures. The reliability of healthcare services really depends on effectiveness of medical equipment. The optimised equipment does not only assist during healthcare procedures, but improve the availability of the equipment while needed. Moreover, it also upkeeps in terms of the safety aspect from any possible failure, which may cause severe harmful against the users and patients. Thus, the optimization of medical equipment performance may upkeep the level of healthcare services and reduce the cost of operations.

The current study on the medical equipment assessment indicated a fundamental understanding of how the assessment contributes toward the effectiveness in delivering healthcare services to the community. The ultimate contribution from this study is that the healthcare institutions are capable of providing better healthcare services to the public by the medical equipment availability, upkeep the safety level by avoiding any failures of equipment that may cause hazards, and prepare and allocate sufficient budget on the expenditure of equipment maintenance and replacement activities.

The review summarised that the assessment of medical equipment conditions will assist the clinical engineers to increase the availability of the equipment in healthcare institutions. Furthermore, the equipment availability will assist the clinical engineering department in healthcare institutions to achieve medical equipment maintenance management effectiveness by prioritising the activity according to the urgency. This prioritising approach may eventually optimise the operational cost and work with available resources.

According to the comparison of the findings in this study with MS2058:2018, the categories classified based on the input parameters extracted from the previous studies and have been correlated with proposed factors in the national standard. However, improvement can be made by adding several factors that covered the function and maintenance requirement categories. The proposal for selecting the input parameters depends on clinical engineers' required objective to overcome the current issue experienced in healthcare institutions. The selected parameters can be practical with the support of accurate existing data.

The expected beneficial output indicated from the assessment technique depends on the experimental input parameters. The output can be generated through systematic processes and professional ways rather than the perception by adapting the appropriate methodological technique. This output can assist clinical engineers in making a proper decision to take the right action to overcome the current issue. Therefore, this study provides recommendations that will be useful for future research:

1) Development of a comprehensive strategic medical equipment maintenance management that covers three main activities, which are PM, CM, and RP. The system shall prioritise the medical equipment at each maintenance activity by measuring the criterion of input parameters proposed in this study. In addition, considering these three activities will provide a comprehensive assessment to the healthcare providers in prioritising resources and proposing appropriate solution in timely manner.

2) Applying ML techniques in assessing the medical equipment condition and reliability. The predictive nature of ML will provide active action in healthcare industry in anticipating medical equipment's failures. Current solutions are based on passive actions which greatly impacted healthcare services providers. Thus, the advancements of ML techniques are deemed to be a practical solution in medical equipment predictive maintenance in mitigating severe failures, optimising resources, improving availability, and upkeeping performance.

3) Imposing adaptive framework on medical equipment reliability based on the functionality of healthcare providers. The capability of ML algorithms in predicting prioritisation of medical equipment maintenance will enable accurate and precise assessment based on the contributing factors of the medical equipment. The framework will be adaptive to the nature of the healthcare institutions' function since the predictive nature of ML algorithm able to find patterns and trends based on the previous scenarios. Thus, specific model can be easily adopted by healthcare providers in ensuring optimised services to community.

Therefore, by applying appropriate techniques that drive the compliance of national standards and statutory requirements, the review provided a new insight in adopting AI and/or ML algorithms which can be aligned in any government's standard in providing better healthcare services.

Author Contributions

AZ and KH designed and developed the study protocol as well as major contributors to the article writing. AZ, AA, and KH performed the identification, screening, eligibility, quality assessment, and information extraction of the articles. MA, SS, and KL checked all the synthesised data and approved the final version to be submitted for publication. All authors have substantial contributed to the article.

This work was supported by the University Malaya Research Grant Faculty Programme (RF010-2018A) and International Funding from Motorola Solution Foundation (IF014-2019).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors would like to express thanks to the Malaysian Ministry of Health for the continuous supports in conducting this study.

Abbreviations

AI, Artificial Intelligence; CM, Corrective Maintenance; ML, Machine Learning; MS2058:2018, Code of Practise for Good Engineering Maintenance Management of Active Medical Devices; PM, Preventive Maintenance; PRISMA, Preferred Reporting Items for Systematic Review and Meta-Analyses; RP, Replacement Programme; WHO, World Health Organisation.

1. Chaudhary P, Kaul P. Factors affecting utilization of medical diagnostic equipment: a study at a tertiary healthcare setup of Chandigarh. CHRISMED J Health Res. (2015) 2:316. doi: 10.4103/2348-3334.165741

CrossRef Full Text | Google Scholar

2. Kavrakov D. Asset vs. Facility Management – Spot the Difference Comparative Analysis of Professional Identity and Functions. Euro FM Insights . (2016). Available online at: https://www.bgfma.bg/content/uploads/2016/01/AMvsFM_FMInsights_DK.pdf (accessed February 01, 2020).

Google Scholar

3. Yik Francis WH, Lai Joseph HK, Yuen PL. Impacts of facility service procurement methods on perceived performance of hospital engineering services. Facilities. (2012) 30:56–77. doi: 10.1108/02632771211194275

4. Shohet Igal M, Lavy S. Healthcare facilities management: state of the art review. Facilities. (2004) 22:210–20. doi: 10.1108/02632770410547570

5. Cesarotti V, Di Silvio B. Quality management standards for facility services in the Italian health care sector. Int J Health Care Qual Assur. (2006) 19:451–62. doi: 10.1108/09526860610687600

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Badnjević A, Gurbeta L, Bošković D, Džemić Z. Measurement in medicine–Past, present, future. Folia Medica Fac Medicinae Uni Saraeviensis . (2015) 15:43–6. Available online at: http://foliamedica.mf.unsa.ba/index.php/FM/article/view/38

7. Wang B. Medical equipment maintenance: management and oversight. Synthesis Lectures Biomed Eng. (2012) 7:1–85. doi: 10.2200/S00450ED1V01Y201209BME045

8. World Health Organization. Development of Medical Device Policies : World Health Organization, Geneva (2011). 39 p.

9. Worm A. Managing The Lifecycle of Medical Equipment . THET (2015).

10. Salim S, Mazlan S, Salim S. A conceptual framework to determine medical equipment maintenance in hospital using RCM method. MATEC Web Conferences. London (2019) 266:02011. doi: 10.1051/matecconf/201926602011

11. Kutor J, Agede P, Ali R. Maintenance practice, causes of failure and risk assessment of diagnostic medical equipment. J Biomed Eng Med Devices. (2017) 02:123. doi: 10.4172/2475-7586.1000123

12. Bahreini R, Doshmangir L, Imani A. Factors affecting medical equipment maintenance management: a systematic review. J Clin Diagnostic Res. (2018) 12:IC1–7. doi: 10.7860/JCDR/2018/31646.11375

13. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. (2009) 6:e1000097. doi: 10.1371/journal.pmed.1000097

14. Martin-Martin A, Orduna-Malea E, Thelwall M, Lopez-Cozar ED. Google scholar, web of science, and scopus: a systematic comparison of citations in 252 subject categories. J Informetr. (2018) 12:1160–77. doi: 10.1016/j.joi.2018.09.002

15. Adriaanse LS, Rensleigh C. Web of science, scopus and google scholar a content comprehensiveness comparison. Electron Libr. (2013) 31:727–44. doi: 10.1108/EL-12-2011-0174

16. Younger P. Using google scholar to conduct a literature search. Nurs Standard. (2010) 24:40–6; quiz 8. doi: 10.7748/ns2010.07.24.45.40.c7906

17. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ . (2021) 372:n71. doi: 10.1136/bmj.n71

18. Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Educ Informat. (2018) 34:285–91. doi: 10.3233/EFI-180221

19. Kovacevic Z, Pokvic LG, Spahic L, Badnjevic A. Prediction of medical device performance using machine learning techniques: infant incubator case study. Health Technol. (2019) 10, 151–5. doi: 10.1007/s12553-019-00386-5

20. Badnjević A, Gurbeta Pokvić L, Hasičić M, Bandić L, Mašetić Z, Kovačević Ž, et al. Evidence-based clinical engineering: machine learning algorithms for prediction of defibrillator performance. Biomed Signal Process Control. (2019) 54:101629. doi: 10.1016/j.bspc.2019.101629

21. Saleh N, Sharawi AA, Abd Elwahed M, Petti A, Puppato D, Balestra G. Preventive maintenance prioritization index of medical equipment using quality function deployment. IEEE J Biomed Health Informat. (2015) 19:1029–35. doi: 10.1109/JBHI.2014.2337895

22. Hernández-López LA, Pimentel-Aguilar AB, Ortiz-Posadas MR. An index to prioritize the preventive maintenance of medical equipment. Health Technol. (2019) 10:399–403. doi: 10.1007/s12553-019-00371-y

23. Jamshidi A, Rahimi SA, Ait-kadi D, Ruiz A. A comprehensive fuzzy risk-based maintenance framework for prioritization of medical devices. Appl Soft Comput. (2015) 32:322–34. doi: 10.1016/j.asoc.2015.03.054

24. Faisal M, Sharawi A. Prioritize medical equipment replacement using analytical hierarchy process. IOSR J Electric Electron Eng. (2015) 10:55–63. doi: 10.9790/1676-10325563

25. Tawfik B, Ouda BK, Abd El Samad YM. A fuzzy logic model for medical equipment risk classification. J Clin Eng. (2013) 38:185–90. doi: 10.1097/JCE.0b013e3182a90445

26. Jarikji Y, Hussein B., Hajj-Hassan M. A quantitative model for replacement of medical equipment based on technical and economic factors. Int J Artificial Organs. (2019) 42:278–85. doi: 10.1007/978-3-030-30874-2_22

27. Aridi M, Hussein B, Hajj-Hassan M, Khachfe HM. A novel approach for healthcare equipments lifespan assessment. Int J Adv Life Sci. (2016) 8:1–15. Available online at: http://www.iariajournals.org/life_sciences/

28. Hamdi N, Oweis R, Abu Zraiq H, Abu Sammour D. An intelligent healthcare management system: a new approach in work-order prioritization for medical equipment maintenance requests. J Med Syst. (2012) 36:557–67. doi: 10.1007/s10916-010-9501-4

29. Hutagalung AO, Hasibuan S. Determining the priority of medical equipment maintenance with analytical hierarchy process. Int J Online Biomed Eng. (2019) 15:107–20. doi: 10.3991/ijoe.v15i10.10920

30. Taghipour S, Banjevic D, Jardine AKS. Prioritization of medical equipment for maintenance decisions. J Operat Res Soc. (2011) 62:1666–87. doi: 10.1057/jors.2010.106

31. Ben Houria Z, Masmoudi M, Al Hanbali A, Khatrouch I, Masmoudi F. Quantitative techniques for medical equipment maintenance management. Europ J Indust Eng. (2016) 10:703–23. doi: 10.1504/EJIE.2016.081017

32. Oshiyama NF, Bassani RA, D'Ottaviano IML, Bassani JWM. Medical equipment classification: method and decision-making support based on paraconsistent annotated logic. Med Biol Eng Comput. (2012) 50:395–402. doi: 10.1007/s11517-012-0888-6

33. Saleh N, Balestra G. Comprehensive framework for preventive maintenance priority of medical equipment. Annu Int Conf IEEE Eng Med Biol Soc. (2015) 2015:1227–30. doi: 10.1109/EMBC.2015.7318588

34. Ismail S, Hussein B, Hajj-Hassan M, Nehme H, Khachfe H. A holistic approach for forecasting medical equipment risks using monte carlo simulation. Int J Adv Life Sci. (2018) 10:1–10. Available online at: http://www.iariajournals.org/life_sciences/lifsci_v10_n12_2018_paged.pdf

35. World Health Organization. Medical Equipment Maintenance Programme Overview: WHO Medical Device Technical Series . Geneva: Department of Essential Health Technologies, World Health Organization (2011).

36. Corciovă C, Andritoi D, Luca C. A modern approach for maintenance prioritization of medical equipment. Maintenance Manage IntechOpen. (2020) doi: 10.5772/intechopen.92706

37. Bahreini R, Doshmangir L, Imani A. Influential factors on medical equipment maintenance management: in search of a framework. J Qual Maintenance Eng. (2019) 25:128–43. doi: 10.1108/JQME-11-2017-0082

38. Wu H, Liu G (editors). Building the quality control system for medical equipments in hospital. In: 2010 International Conference on Management and Service Science . London (2010). doi: 10.1109/ICMSS.2010.5576119

39. Mahfoud H, El Barkany A, El Biyaali A. Medical maintenance performance monitoring: a roadmap to efficient improvement. Int J Productiv Qual Manage. (2017) 22:117–40. doi: 10.1504/IJPQM.2017.085850

40. Jamshidi A, Rahimi SA, Ait-Kadi D, Bartolome AR (editors.). Medical devices inspection and maintenance; a literature review. In: Proceedings of the 2014 Industrial and Systems Engineering Research Conference . Wuhan: Institute of Industrial Engineers. (2014).

41. Diani CA, Rock A, Moll P. An evaluation of the effectiveness of a risk-based monitoring approach implemented with clinical trials involving implantable cardiac medical devices. Clin Trials. (2017) 14:575–83. doi: 10.1177/1740774517723589

42. Kementerian Kesihatan Malaysia. Laporan Tahunan Kementerian Kesihatan Malaysia 2018 . In: Centre HI, editor. Malaysia: Kementerian Kesihatan Malaysia (2018).

43. KPJ Healthcare Berhad. Financial Report 2019 , Montreal (2019).

44. World Health Organization. Malaysia Health System Review. Health Systems in Transition . Manila: WHO Regional Office for the Western Pacific (2012).

45. Ministry of Health Malaysia. Health Facts 2018 (Reference Data for 2017) . In: Planning Division HIC, editor. Malaysia: Ministry of Health Malaysia (2018).

46. Ministry of Health Malaysia. Health Facts 2019 (Reference Data for 2018) . In: Planning Division HIC, editor. Malaysia: Ministry of Health Malaysia (2019).

47. Ministry of Health Malaysia. Health Facts 2020 (Reference Data for year 2019) . In: Planning Division HIC, editor. Malaysia: Ministry of Health Malaysia (2020).

48. Ministry of Health Malaysia. Strategic Framework of the Medical Programme, Ministry of Health Malaysia (2021 – 2025) . Hospital Management Unit MDD, editor. Malaysia: Ministry of Health Malaysia (2020).

49. Kementerian Kesihatan Malaysia. Polisi Perkhidmatan Rawatan Harian di Hospital-Hospital Kementerian Kesihatan Malaysia . Perubatan BP, editor. Malaysia: Kementerian Kesihatan Malaysia (2016).

50. Ministry of Health Malaysia. Laporan Audit Dalam Tahun 2020 . Malaysia: Ministry of Health Malaysia (2020).

51. Department of Standards Malaysia. Code of Practice for Good Engineering Maintenance Management of Active Medical Devices (Second revision) . In: Standard M, editor. Kuala Lumpur: MS 2058:2018: Department of Standards Malaysia (2018).

52. Department of Standards Malaysia. Standards of Malaysia Act 1996 (Act 549) . Cyberjaya: Department of Standards Malaysia (2012). p. 1–35. Available online at: https://www.jsm.gov.my/act-and-regulation

53. Medical Device Authority. Medical Device Act 2012 (Act 737) . Kuala Lumpur: Percetakan Nasional Malaysia Berhad (2012). p. 1–49. Available online at: https://www.mda.gov.my/doc-list/legislation.html

54. Medical Device Authority. Medical Device Authority Act 2012 (Act 738) . Kuala Lumpur: Percetakan Nasional Malaysia Berhad (2012). p. 1–24. Available online at: https://www.mda.gov.my/doc-list/legislation.html

Keywords: medical devices, biomedical equipment, performance evaluation, maintenance management, assessment, prediction

Citation: Zamzam AH, Abdul Wahab AK, Azizan MM, Satapathy SC, Lai KW and Hasikin K (2021) A Systematic Review of Medical Equipment Reliability Assessment in Improving the Quality of Healthcare Services. Front. Public Health 9:753951. doi: 10.3389/fpubh.2021.753951

Received: 05 August 2021; Accepted: 31 August 2021; Published: 27 September 2021.

Reviewed by:

Copyright © 2021 Zamzam, Abdul Wahab, Azizan, Satapathy, Lai and Hasikin. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Khairunnisa Hasikin, khairunnisa@um.edu.my ; Muhammad Mokhzaini Azizan, mokhzainiazizan@usim.edu.my

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

• US: 1.800.472.6477
• EU: +353 21 212 8530

QA/RA Consulting, Auditing & Training

Let's get started

• November 2, 2023

Does Your Medical Device CER Meet EU MDR Requirements? Creating a Solid Clinical Evaluation Process

In this guide:

Stages in the clinical evaluation process Defining the scope and drafting a plan Making sure CER evaluators are qualified Creating a literature review protocol Identifying data needed to fulfill plan requirements Crafting a literature search and review strategy Choosing the appropriate data Is your data valid and relevant? Appraising the clinical data Analyzing your datasets and drawing conclusions Clinical data analysis Alignment between clinical evaluation, IFU, and risk management Are additional clinical investigations needed? Writing your CER and when to update it Compiling the clinical evaluation report Creating your EU CER template Create a CER checklist How often your CER should be updated

Medical device regulatory professionals have been grappling with tighter requirements for clinical data to support clinical evidence since MEDDEV 2.7/1 Rev. 4 was released in mid-2016. Now that the European Medical Device Regulation (2017/745) is nearing implementation, clinical evaluation reports (CERs) have taken on new urgency.

Regardless of whether you are making updates to existing CERs or building one for a product launch, you want to make sure that you don’t experience any nasty surprises during your Notified Body audit and technical documentation review.  The first step is to create a robust clinical evaluation plan.

As you probably know, medical device manufacturers can fulfill clinical evidence requirements using one or more of the following:

• Premarket clinical studies (the mostoption, but necessary for new technology/indications)
• Literature searches for existing clinical studies for equivalent devices (
• Compliance with harmonized standards (e.g., IEC 60601-1 on electrical safety)
• Data generated from PMS activities and postmarket clinical studies, such as postmarket clinical follow-up (PMCF)
• Clinical investigations
• Postmarket clinical follow-up

Stages in the Medical Device Clinical Evaluation Process

Before we go into detail on the process, here’s an overview of the various stages outlined in MEDDEV 2.7/1 Rev. 4 and where you will find guidance on each stage.

Clinical Evaluation Stage 0: Defining the Scope and Drafting a Clinical Evaluation Plan

Published literature is a key component of the clinical data gathered by most companies. While it may be tempting to dig in and start doing literature searches straight away, it’s important that you understand what is needed in the first place. This “scoping” process is vital because you will need to explain and defend it to your Notified Body, and you will need to replicate it in the future.

Your clinical evaluation plan will define the extent of information gathered based on the General Safety and Performance Requirements in Annex I of the MDR. MEDDEV 2.7/1 Rev. 4 defines numerous aspects that should be considered in a thorough clinical evaluation plan. You can find much more detail on the elements to be considered during Stage 0 in Section 7 of the MEDDEV, which delineates between new medical devices and those that already have CE Marking.

MEDDEV 2.7/1 Rev. 4 states : Clinical evaluation is necessary and important because it ensures that the evaluation of safety and performance of the device is based on sufficient clinical evidence throughout the lifetime that the medical device is on the market.

Here are some of the aspects that need to be included in your CER scoping process. Some of these issues apply only to “new” devices getting CE Marking for the first time.

• Device description – see Appendix A3 of the MEDDEV
• CE Marking status of the device
• Where the device is marketed outside Europe
• Design features, names, models, sizes, components, etc.
• Intended use of the device and special indications or target populations
• Risk management documentation
• Current device  State Of The Art
• Data sources and types to be used
• Information needed for demonstrating  Equivalence (new devices only)
• Design, manufacturing, materials, or labeling changes (devices with CE Marking)
• Any newly emerged clinical concerns (devices with CE Marking)
• New data generated from PMS (devices with CE Marking)

Of course, the amount of data deemed necessary to meet sufficient clinical evidence and the General Safety and Performance Requirements will also be determined by the nature of the device, its stage in the life cycle, and its safety record.

Article 1(a) of Annex XIV in the MDR provides additional detail about what the clinical evaluation plan should include. You’ll want to study this in addition to Section 7 of the MEDDEV before crafting your plan.

Making Sure CER Evaluators Are Qualified

Before you dive into planning, you’ll also want to know that the MEDDEV sets some pretty specific guidelines for who can take on this important task. Evaluating clinical data is serious business, and regulators want to ensure that the people performing the evaluation are well qualified to do so. Section 6.4 of the MEDDEV outlines the basic requirements. Evaluators should have an appropriate college degree plus 5 years of professional experience or 10 years of experience if a degree is not a prerequisite. In addition, Section 6.4 goes on to say that evaluators should have knowledge of:

• Research methodology
• Information management
• Regulatory requirements
• Medical writing
• Application of the specific device technology
• Diagnosis and management of the conditions to be diagnosed or managed by the device

These qualifications will need to be documented and you can be certain that your Notified Body will review documentation related to reviewer qualifications.

Creating an EU CER Literature Review Protocol and Reviewing Medical Device Clinical Data

Once you have created a robust clinical evaluation plan for your medical device(s), it’s time to get busy figuring out what data you need, drafting a protocol, and then taking stock of that data. This is considered Stages 1 and 2 of the CER process and the specifics can be found in MEDDEV 2.7/1 rev. 4, as shown below.

Clinical Evaluation Stage 1: Identifying Data Needed to Fulfill the Requirements of Your Plan

There are two broad categories of clinical data.

1 – Data  not   generated by your company  (retrieved from online literature searches and other offline data)

2 – Data generated by your company, which can include:

• Premarket clinical investigations, including bench testing reports
• Postmarket data gathered from risk management and PMS activities, including PMCF studies, device registries, and more

Crafting a CER Literature Search and Review Strategy

For many companies, the data retrieved from literature searches will represent most, if not all, of the data they collect. That’s why it is so critically important that you develop a literature search strategy that is robust and can be replicated during subsequent updates to your CER. The output of your search and review should obviously include literature on your device and any identified  equivalent device , plus a review of the current state of the art. Your literature search protocol should include the following elements (for more specifics on this, see Appendix A5 in MEDDEV 2.7/1 Rev. 4):

• Sources of data you will use (e.g., MEDLINE/PubMed, Embase, Google Scholar, ResearchGate, internet searches, etc.)
• The methodology you plan to use for searches
• Exact search terms and parameters (e.g., dates) used to search scientific databases and the internet
• Your specific selection or exclusion criteria along with justification for each
• How you will address duplication of data from multiple sources
• How you will ensure data integrity (e.g., QC methods or second reviewers)
• How you will appraise each data source and its relevance to your device
• How you will go about analyzing and processing the data

You should think of your plan as you would a standard operating procedure (SOP) or a detailed instruction. As an example, if you were to leave the company a year from now, your successor should be able to read your protocol and understand exactly what was done during the previous update.

Make sure you treat the literature searches related to device equivalence as a separate activity from the searches on current state of the art. Separately define and track search terms used, database sources, date parameters, etc. Also, be cognizant that your idea of “state of the art” may differ from reality – here’s  why . Before finalizing your protocol, test it. Identify known papers relevant to your device or current state of the art and then test those search parameters to make sure those known papers appear in your search results.

Review It All: The Favorable…and the Not So Favorable

Your literature searches should be extremely thorough and encompass a breadth of search criteria. They also need to be documented in detail so the results can be independently verified and replicated. Your selection of literature should be objective (the good and the bad) and justifiable. Nearly every published study has citations that may lead you to additional relevant data. Again, this is where planning and staying within your device search parameters become really important, because you can very quickly wander down a virtual rabbit hole and not be able to replicate how you got there.

While most publications are available in English (e.g., 93% on MEDLINE), make some attempts to do basic searches in French, German, or Japanese using online translation tools. Obviously, if you work for a larger company that has offices in Europe or Asia, getting help from native-speaking colleagues would be ideal. While the odds are good that you will find what you need in English, you might uncover valuable data that can be translated and further appraised if needed.  Also, if you are updating an existing CER, we recommend that you start your search a few weeks before the end date of previous search to ensure you don’t miss anything that may have been added at the very end of your previous search period but had not yet been indexed online.

Keep in mind that some data may be publicly available but not easily accessible online. For instance, the label and IFU of a competitive device (if using one) may yield useful information. Also, presentations made at industry conferences may provide information that is not available (or has not yet been published) online.

Clinical Evaluation Stage 2: Choosing the Appropriate Data Sets Also Requires a Plan

During Stage 0, you came up with a clinical evaluation plan focused on  determining which data  you need and from what sources. In Stage 2, your focus will turn to  choosing the right data  among the data sets you have found. This requires yet more planning. Your data appraisal plan should address:

• Criteria for determining the quality, relevance, and validity of the data
• How you will weigh the data related to favorable and unfavorable results

According to Section 9.2 of MEDDEV 2.7/1 Rev. 4, the criteria you select should “reflect the nature, history and intended clinical use of the device.” This is something you must document and justify based on current state of the art.

Is Your Data Valid and Relevant?

So you found plenty of data in Stage 1 that seem relevant to your device and its intended purpose. But how do you know if the quality is good? At first glance, you don’t. Sure, a paper published in  The Lancet  is instantly credible, but the quality of a publication cannot serve as the sole rationale for selection. Ultimately you need to dive headfirst into the pool and do investigative research of your own to appraise the quality of the data.

Doing this requires some skill, because it is on you to evaluate the methodology used to collect the data and therefore determine its scientific merit. As part of this appraisal process you will also need to weigh the contribution of each data set to the overall clinical evaluation. Section 9.3 of the MEDDEV contains seven pages of advice on how to evaluate the methodological quality and scientific validity of data sets.

A Higher Bar for Medical Device Equivalency Under the MDR

Many companies use equivalency claims to avoid having to conduct redundant pre- or postmarket clinical studies that prove safety and performance. This part of the clinical evaluation process is not new. What is new is the level of scrutiny those comparative evaluations will endure. The clinical evaluation requirements in the existing Medical Devices Directive (93/42/EEC) and MEDDEV 2.7/1 Rev. 3 largely favored device equivalency but did not define it. Thus, some companies took a liberal view of “equivalent.” When MEDDEV 2.7/1 Rev. 4 was released in mid-2016, it gave manufacturers a lot less latitude for determining which devices could be considered equivalent. In fact, Appendix A1 in Rev. 4 is quite clear about the clinical, technical, and biological characteristics your device must have in common with an “equivalent” device.

The MDR furthers tightens the screws for Class III and implantable devices, requiring a more in-depth assessment and making it more challenging to leverage competitor data for new devices. That’s because Article 61, Section 5 of the MDR requires manufacturers of such devices to have access to the full technical documentation of the competitive device(s). It also instructs Notified Bodies to ask for proof that you have a contract in place granting you permanent access to that technical documentation. Good luck with that.

Appraising the Medical Device Clinical Data

With an appraisal plan created and a grip on how to execute it, you can begin the hard work of appraising the data you have found. This appraisal must be done based on the complete text of publications you find, not just by reading the abstracts or summaries. For each document you appraise, you are required to document your appraisal of it to the point that it could reasonably be reviewed by others. The appraisal results should also support conclusions you are making about clinical safety and clinical performance of the finished device (e.g., citing non-device-related literature would be ranked low for appraisal).

Appendix A6 in MEDDEV 2.7/1 Rev. 4 can be helpful in performing your appraisal of data. It provides some examples of red flags that should make you pause, including clinical data that:

• Lacks basic information such as the methods used, number of patients, identity of products, etc.
• Has data sets that are too small to be statistically significant
• Contains data that applies improper statistical methods
• Employs studies that lack adequate controls
• Has an improper collection of mortality and serious adverse event data
• Depicts a misrepresentation by the authors

The issue of evaluating statistical methods and significance intimidates many RA/QA professionals. If you feel uncomfortable about your “stats skills” for conducting statistical analysis or reviewing statistical information, hire some outside expertise to evaluate these specific aspects. Better to invest in doing that now rather than have a Notified Body reviewer challenge your appraisal later.

Analyzing Your Medical Device Clinical Datasets and Drawing Conclusions

Anyone who has ever researched and compiled an entire European clinical evaluation report knows that the devil is in the details. Finding appropriate clinical data is not necessarily the most challenging aspect of the process – it’s the appraisal and analysis of that data that causes regulatory heartburn. 

Many professionals interchange appraisal and analysis, but those are actually two distinct steps in the process. When you appraise data, you are looking to make sure it has statistically significant data sets, uses proper statistical methods, has adequate controls, and properly collects mortality and/or serious adverse event data.

During the analysis stage (Stage 3) you will really dig in and conduct a comprehensive assessment to determine if the data you have found meets clinical safety requirements, clinical performance requirements, and General Safety and Performance Requirements (GSPR) of the EU Medical Device Regulation (MDR). You will evaluate the following:

• Is the performance bench testing
• Is the risk-benefit ratio appropriate based on the intended purpose of the device?
• Can the device actually achieve all performance claims made by the manufacturer?
• Are the materials (e.g., IFU) supplied by the manufacturer adequate to describe the intended purpose and mitigate risk?

Clinical Evaluation Stage 3:  Clinical Data Analysis

During Stage 3 you are expected to analyze the data to ensure the clinical evaluation demonstrates that any risks are minimal and acceptable. You also need to consider all aspects of the device’s intended purpose. You’ll find more detail on this in Section 10.2 of MEDDEV 2.7/1 Rev. 4, and in Annex AVII. You will identify gaps related to:

• Understanding the interaction between the device and body
• The completeness of the data available
• Type and adequacy of patient monitoring
• Number and severity of adverse events
• Severity and history of the condition being treated or diagnosed
• Current standards of care
• Other factors

Data from the literature you have appraised is often put into Excel tables to be analyzed. It’s a convenient way to compare different study details, patient populations, endpoints, adverse events, etc. This can end up being a sizable amount of data unto itself, but certain aspects can be parsed into “bite-sized” tables focused on particular issues. While spreadsheets are not especially wonderful for narrative explanations, they are useful as quick comparative tools and are extremely helpful in noting differences between studies when writing the summary.

Alignment Between Clinical Evaluation, IFU, and Risk Management

Your analysis should also examine the alignment between the clinical evaluation, labeling/instructions for use (IFU), and the risk management file, as well as the current  state of the art . Reviewers need to pay very close attention to make sure that, for example, the IFU and promotional materials are harmonized with regard to medical conditions and target populations. This analysis also needs to be consistent with the appraisal you conducted during Stage 2. Here’s what Appendix A7.1 of MEDDEV 2.7/1 Rev. 4 has to say about it:

“The information materials supplied by the manufacturer (including label, IFU, available promotional materials including accompanying documents possibly foreseen by the manufacturer), should be reviewed to ensure they are consistent with the relevant clinical data appraised in Stage 2 and that all the hazards, information on risk mitigation, and other clinically relevant information have been identified appropriately.”

It is expected that you will conduct the analysis with input from your risk management files and appropriate standards such as IEC 60601-1 (electrical safety) and EN 62366 (usability). The goal, of course, is to demonstrate that any risks associated with the intended purpose of your device are acceptable when weighed against the benefits it offers the patient or user.

Are Additional Clinical Investigations Needed?

How much is enough? There is no definitive rule that determines whether you have collected enough clinical data to meet the General Safety and Performance Requirements (GSPR) of the EU Medical Device Regulation (MDR). You need to conduct a detailed gap analysis and come to your own conclusion about whether supplemental clinical investigations will be required. At this time, you should also determine whether there are residual risks and uncertainties. This might include factors related to rare complications, long-term performance, or safety under widespread use. You’ll find more information on this in Appendix A2 of MEDDEV 2.7/1 Rev. 4.

Writing Your EU MDR Clinical Evaluation Report and When to Update It

You’ve done a lot of work to get this far. Writing a medical device clinical evaluation report (CER) is the culmination of a monumental effort to conduct literature searches, find/review literature, and/or conduct original clinical investigations. Data must be sourced, appraised, analyzed, and then summarized into your CER. This final process – writing the CER itself – is commonly referred to as Stage 4.

Clinical Evaluation Stage 4:  Compiling the CER

Creating Your EU CER Template

Because the contents of a clinical evaluation report vary according to the nature and history of the device being evaluated, neither MEDDEV 2.7/1 Rev. 4 nor the EU MDR provide a detailed CER template. However, Appendix A9 of the MEDDEV does provide nearly six pages of guidance on what the structure of your CER should look like and what content it should contain. Here’s the basic outline.

1 – Summary

2 – Scope of the clinical evaluation

3 – Clinical background, current knowledge, and state of the art

4 – Device under evaluation:

• Type of evaluation
• Demonstration of equivalence, if applicable
• Clinical data generated and held by manufacturer
• Clinical data generated from literature searches
• Summary and appraisal of clinical data
• Analysis of the clinical data

5 – Conclusions

6 – Date of next clinical evaluation

7 – Dates and signatures

8 – Qualification of the responsible evaluators

9 – References

Create a CER Checklist: Do You Have Everything Covered?

You’ve done all the hard work, but are you sure you didn’t forget anything? The checklist found in Appendix A10 should help. Here’s an abbreviated version of it. (Note that this is only a partial list, to give you a flavor of what your CER checklist should include.)

• Is the report understandable to a third party and does it provide sufficient detail?
• Is all data generated, mentioned, and summarized in the report?
• If claiming equivalence, are differences adequately disclosed and explained as to why you don’t expect them to affect safety and performance?
• Has the latest PMS/PMCF data been taken into account and summarized?
• Is current state of the art explained and substantiated?
• Are undesirable side effects and the risk/benefit profile acceptable when compared to state of the art?
• Is conformity to MDR General Safety and Performance Requirements stated?
• Do informational materials supplied correspond with contents of the report?
• Does the report identify all residual risks or uncertainties that should be addressed in PMS/PMCF studies?
• Is the report dated and qualification of the evaluators included?

If the CER covers several models, sizes, settings, or situations, are the conclusions correct for:

• All devices, sizes, models, and settings?
• Every medical indication?
• The entire target population and all intended users?
• The duration of product use, including repeat exposure?

How Often Your EU CER (Clinical Evaluation Reports) Should Be Updated

Section 6.2.3 of MEDDEV 2.7/1 Rev. 4 provides guidance to manufacturers on how often to update clinical evaluations. It says that the “manufacturer should define and justify the frequency” of CER updates. Typically, this is done in concert with your Notified Body audit and certificate renewal, but that predefined schedule can be tossed out the window if your postmarket surveillance activities uncover new risks.

Shown below is an example of how you could rationalize the frequency of updates to your CER. This sample shows a simple evaluation table for an ECG machine that is well established on the market and has decent clinical history. You can, of course, create your own rating system with more factors, and you could weight these factors as well. We’ve kept the table simple in order to illustrate the point that your NB wants to see that you have a systematic approach to determining your CER update schedule.

As you can see in this example, even though this device is an established ECG with good clinical history, due to its inherently complexity, where it is used on the body, and the target population on which it is used, updating the CER every few years is still recommended.

Regardless of your proposed rationalization for the update schedule, you  must  update the CER after getting new information from your postmarket surveillance activities that might change your current evaluation. For example, safety reports, newly published literature, or PMCF studies may uncover previously unknown safety concerns. Data from these sources must be to evaluated because it may change your risk/benefit profile. Remember that Annex XIV, Part B of the MDR mandates that your PMCF plan specify the “methods and procedures for proactively collecting and evaluating clinical data.”

Your CER Is a Critical Component of Your Technical Documentation

Along with your risk management documentation and postmarket surveillance plan, the clinical evaluation report is a centerpiece of the technical documentation needed for CE Marking. While we think of the CER primarily as a regulatory exercise, it’s important to remember that the ultimate goal is the advancement of patient safety. Looking at it through that lens will help you focus on the right things and create a CER in full compliance with MEDDEV 2.7/1 Rev. 4 and the EU MDR (2017/745).

Advance Your Knowledge of Clinical Evaluation Reports

If you found this article to be informative and you want to take the next step in advancing your knowledge of all things CER, consider our  EU CER training class . Our consultants are also available to help you with  EU CER development and gap analysis .

Our team is here to help. Contact us online

Get answers right now. call.

US Office Washington DC

1.800.472.6477

EU Office Cork, Ireland

+353 21 212 8530

Search by Topic

• All Category
• CAPA and Root Cause Analysis
• Clinical Evaluation Reports
• Combination Products
• Complaint Handling and PMS
• Design Control
• EU IVDR (2017/746)
• EU MDR (2017/745)
• FDA 21 CFR Part 820
• FDA 510(k) Submissions
• ISO 13485 QMS Compliance
• ISO 14971 Risk Management
• Lab Developed Tests
• Pharmaceuticals
• QMS Auditing
• Software and Cybersecurity
• Supplier Management
• Uncategorized

UNITED STATES 1055 Thomas Jefferson St. NW Suite 304 Washington, DC 20007 Phone: 1.800.472.6477

4 Emmet House, Barrack Square Ballincollig Cork, Ireland Phone: +353 21 212 8530

• Consulting & Auditing

Find a Training Course

• International Offices

Consulting & Auditing

• Medical Device RA/QA
• MDSAP, ISO 13845, QSR, & EU MDR
• Pharmaceutical RA/QA
• Connected Medical Technology
• Performance Excellence
• ISO 9000 & Related Standards

© Oriel STAT A MATRIX. All Rights Reserved.

• Methodology
• Open access
• Published: 10 September 2024

Developing a process for assessing the safety of a digital mental health intervention and gaining regulatory approval: a case study and academic’s guide

• Rayan Taher 1 ,
• Charlotte L. Hall 2 ,
• Aislinn D Gomez Bergin 2 , 3 ,
• Neha Gupta 4 ,
• Clare Heaysman 5 ,
• Pamela Jacobsen 6 ,
• Thomas Kabir 7 ,
• Nayan Kalnad 4 ,
• Jeroen Keppens 8 ,
• Che-Wei Hsu 9 ,
• Philip McGuire 10 ,
• Emmanuelle Peters 11 ,
• Sukhi Shergill 12 ,
• Daniel Stahl 13 ,
• Ben Wensley Stock 14 &
• Jenny Yiend   ORCID: orcid.org/0000-0002-1967-6292 1  

Trials volume  25 , Article number:  604 ( 2024 ) Cite this article

Metrics details

The field of digital mental health has followed an exponential growth trajectory in recent years. While the evidence base has increased significantly, its adoption within health and care services has been slowed by several challenges, including a lack of knowledge from researchers regarding how to navigate the pathway for mandatory regulatory approval. This paper details the steps that a team must take to achieve the required approvals to carry out a research study using a novel digital mental health intervention. We used a randomised controlled trial of a digital mental health intervention called STOP (Successful Treatment of Paranoia) as a worked example.

The methods section explains the two main objectives that are required to achieve regulatory approval (MHRA Notification of No Objection) and the detailed steps involved within each, as carried out for the STOP trial. First, the existing safety of digital mental health interventions must be demonstrated. This can refer to literature reviews, any feasibility/pilot safety data, and requires a risk management plan. Second, a detailed plan to further evaluate the safety of the digital mental health intervention is needed. As part of this we describe the STOP study’s development of a framework for categorising adverse events and based on this framework, a tool to collect adverse event data.

We present literature review results, safety-related feasibility study findings and the full risk management plan for STOP, which addressed 26 possible hazards, and included the 6-point scales developed to quantify the probability and severity of typical risks involved when a psychiatric population receives a digital intervention without the direct support of a therapist. We also present an Adverse Event Category Framework for Digital Therapeutic Devices and the Adverse Events Checklist—which assesses 15 different categories of adverse events—that was constructed from this and used in the STOP trial.

Conclusions

The example shared in this paper serves as a guide for academics and professionals working in the field of digital mental health. It provides insights into the safety assessment requirements of regulatory bodies when a clinical investigation of a digital mental health intervention is proposed. Methods, scales and tools that could easily be adapted for use in other similar research are presented, with the expectation that these will assist other researchers in the field seeking regulatory approval for digital mental health products.

Peer Review reports

The field of digital mental health interventions (DMHIs) has followed an exponential growth trajectory in recent years [ 1 ]. DMHIs typically involve mental health interventions, such as cognitive behavioural therapy, delivered via digital technologies, such as smartphones, and can either be completed as self-directed interventions or blended alongside synchronous (e.g., face-to-face or videoconference) or asynchronous (e.g., email or text message) clinical support [ 2 ]. The main benefit of these interventions is delivering evidence-based care to a large number of people with limited clinical resources [ 3 ]. While the evidence base has increased significantly, the adoption of these interventions within health and care services has been slowed by several challenges, including a lack of knowledge from researchers regarding how to navigate the pathway for mandatory regulatory approval. In the UK, DMHIs must meet the standard of evidence set by the National Institute of Health and Care Excellence (NICE) for adoption within the National Health Service (NHS) [ 4 ]. For DMHIs that are developed to diagnose, prevent, monitor, treat, or alleviate a mental health condition, this may include regulation as a “Software as a Medical Device” (SaMD) by the Medicines and Healthcare products Regulatory Agency (MHRA) [ 5 ]. The regulatory process ensures that devices used within the health and social care context are safe and effective.

In some cases, research will involve digital therapeutics that are already in use and carry a CE or UKCA mark. In this case, the therapeutic’s safety and effectiveness has already been established (and is maintained either through self-certification by the manufacturer or, for higher risk devices, through the use of a “Notified Body”: a government-approved organisation that ensures the device continues to conform to the required standards). However, early-stage digital therapeutics will not yet bear a CE/UKCA mark and are therefore required to obtain a specific form of regulatory approval from the MHRA (called “Notification of No Objection”; NoNO) before being used in research, in addition to the usual ethical approvals [ 6 ]. The NoNO regulatory process requires that safety and effectiveness data collection are the primary purpose of a clinical investigation, with the overall aim being to establish whether the benefits of the device outweigh its risks. This places a number of constraints and requirements upon how researchers design their investigations and write their protocols, the most obvious being that rigorous safety assessment is paramount. The present paper is intended to help academics who are interested in digital therapeutics, but unfamiliar with medical device safety assessment, to navigate a course through this complex regulatory field.

Although the research proposal for which NoNO is sought will, as already explained, need to have safety as a primary outcome, obtaining NoNO also requires the research team to demonstrate the safety of their device before the proposed investigation can be approved [ 6 ]. To understand this apparent contradiction, it is crucial to appreciate that safety assessment is considered an inherently iterative process: preliminary safety data must be presented in order to justify collecting more detailed safety data. This can be done by providing a summary of the existing device safety information using all possible sources (e.g. prototypes, user testing, pilot or feasibility data, qualitative information); a risk management plan (identifying all possible hazards, their potential impact and mitigations) and a detailed plan for safety assessment in the proposed clinical investigation (such as collecting and assessing any untoward medical occurrences [ 7 ], usually called adverse events (AEs)) [ 6 ].

However, researchers investigating DMHIs face specific challenges when proposing a safety assessment plan. Notably, MHRA guidance was developed in consideration of medical devices used in clinical contexts such as surgery and pharmacological interventions and was not designed to accommodate the unique safety considerations relevant to DMHIs. Additionally, the guidelines used in research for assessing the safety of DMHIs are borrowed from the medical and pharmaceutical fields, such as the “International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use—Good Clinical Practice” (ICH-GCP) guidelines [ 8 ]. These medical guidelines do not transfer well to assessing the safety of both digital and non-digital psychological interventions because of the fundamental differences between pharmacological and psychological approaches to treatment [ 8 ]. For example, biological responses to medicines usually occur rapidly and can be objectively measured, whereas psychological responses to therapy rely heavily on patients’ self-reported symptoms, can be hard to disentangle from other contextual factors, and intervention effects can take days, weeks or even months to emerge. As others have also identified, using medical definitions and processes to assess the safety of non-medical interventions such as DMHIs and behaviour change interventions can be unhelpful. It can overcomplicate the process of safety assessment, and lead to missing important harms [ 8 , 9 , 10 ].

These concerns have already been raised in trials assessing the safety of behaviour change interventions [ 8 ]. For example, in a qualitative study on recording harms in RCTs for behaviour change interventions, experts emphasised the need for harm recording to be proportionate and focused on harms that are plausibly linked (i.e. related) to the intervention under study [ 34 ]. It is likely that medical processes are being used to assess the safety of DMHIs, because there are no regulatory or standard safety assessment processes in place for face-to-face mental health interventions [ 9 ]. This is surprising given that most adverse events/side effects are common to both face-to-face mental health interventions and DMHIs (e.g. short-term deterioration, novel symptoms, and non-response) [ 10 , 11 , 12 ]. The one area that differs is, of course, technical and device-related harms.

Two recent reviews found that the identification and categorisation of AEs in DMHI trials was inconsistent and often inadequate [ 3 , 11 ]. This was similar to findings of a review on safety assessment in non-pharmacological psychological, behavioural and lifestyle interventions [ 8 ]. It is essential that harmonised standards tailored specifically to the needs of DMHIs are developed. Support mechanisms can then be implemented to assist manufacturers and researchers to understand and adhere to these guidelines . In the absence of these, the purpose of this paper is to share a worked example of how our clinical trial team successfully applied and received the MHRA’s NoNO for STOP (Successful Treatment for Paranoia).

STOP is a mobile app DMHI that uses Cognitive Bias Modification for paranoia (CBM-pa) to reduce symptoms of paranoia [ 12 , 13 ]. STOP consists of 12 weekly sessions of about 40 min each. In each session, the user is presented with 40 ambiguous scenarios that could be interpreted in a paranoid manner. Users are then guided to reevaluate each scenario in a non-paranoid way by completing words and answering questions designed to suggest alternative meanings. The goal is to gradually retrain the brain to assume non-paranoid meanings of ambiguous situations that occur in daily life, which has been shown to reduce paranoid symptoms. More information about STOP and its development is provided elsewhere [ 13 ]. Using STOP as our example, we aimed to provide valuable insights to other research teams undertaking clinical investigations of DMHIs, particularly those requiring regulatory (e.g., MHRA) approval and guidance in the assessment of safety in DMHI research.

Participants

The work presented in this paper was collaboratively completed by academics and clinicians in the field of digital mental health, an expert regulatory consultant, device manufacturers (Avegen), a clinical trials unit at a university, and representatives from an organisation working with experts by experience (the McPin Foundation). Participants varied at each stage; more detail is provided below per task. See Appendix A for the full list of participants.

To assess the safety of STOP (ISRCTN17754650) and obtain the MHRA’s NoNO the STOP team needed to achieve two main objectives:

Demonstrate existing safety.

Evaluate safety within the proposed research for which approval was being sought.

Demonstrating existing safety

This objective was achieved by completing three separate tasks; an empirical feasibility study; relevant literature searches and the creation of a comprehensive Risk Management Plan.

Feasibility study

The research team needed to present current safety data relevant to STOP such as previous publications, feasibility or pilot studies from the same or similar devices/ interventions. To achieve this, the team referred to a previously conducted study that had assessed the intervention’s feasibility and safety as a desktop intervention [ 12 ]. The feasibility study included two arms: treatment (CBM-pa which is a 6-session version of the therapeutic intervention used in STOP but delivered using a desktop computer) and an active control (a version of the same 6 session desktop programme with the same design and format as CBM-pa, except the content was neutral and should not trigger paranoid thoughts) [ 12 ]. CBM-pa works in the same way as STOP (see “ Background ”) by presenting users with a scenario that could be interpreted in a paranoid way and then, using word tasks and questions, helping participants to interpret the scenario in a nonparanoid, benign way [ 12 ]. Sixty-three outpatients with clinically significant paranoia participated in the feasibility study and were randomised to either the treatment or control group [ 12 ].

The feasibility study assessed safety by measuring whether presenting participants with these potentially paranoia-inducing scenarios was distressing or provoking for them, using visual analogue scales (VAS) completed before and after every session and measuring state anxiety, sadness, paranoia and friendliness. These data were used as proxy safety data relevant to STOP because STOP uses the same content and procedures as CBM-pa but delivered in a different format (mobile app vs. desktop) and the sample size of the feasibility studies are usually small [ 12 , 13 ].

Literature searches and regulatory databases

Second, three members of the research team (RT, CH, JY) conducted two literature searches to identity any published safety data that might be relevant to STOP or any equivalent intervention. These literature reviews are different to those used in academia and thus follow a different structure [ 14 ]. The team worked with an independent regulatory consultant on these to make sure we followed industry and regulatory standards. See Appendix B for more details around the methodology used in these two separate literature reviews. As part of this review, FDA databases were also searched for similar devices and any reported adverse events.

Risk management plan

The risk management plan was created collaboratively by key members of the STOP trial academic team and other project stakeholders, including members from a Lived Experience Advisory Panel, members of the software manufacturer and a regulatory consultant (see Appendix A, column 1, for full details).

Under this step, and by using the knowledge that arose from the feasibility study and the review, we developed a risk management plan. This step is required to demonstrate to regulators that the research team has listed all possible hazards, documented what harms might result from each hazard and identified actions or changes that will mitigate every risk entry as far as possible. Regulators require a comprehensive risk analysis specific to the product, showing a clear understanding of the following stepwise process: hazard, harm, initial risk rating, risk controls/ mitigations, revised risk ratings, identification of residual risks and final demonstration that the expected product benefits outweigh the identified residual risks, which should have been reduced as far as possible. In STOP’s case, a residual risk matrix (likelihood × severity) demonstrated that there were no residual medium or high risks.

The risk management plan described above was implemented by carrying out the following activities:

Hazard identification

To develop a risk management plan, the team first needed to develop a list of all potential hazards that participants taking part in the trial could be exposed to and articulate the harms that could result. Based on ISO 14971, which is a standard for risk management for medical devices, hazards are defined as “a potential source of harm” and harms are defined as “injury or damage to the health of people, or damage to property or the environment” [ 15 ]. This was done during a 1-h consensus meeting with an expert regulatory consultant, two representatives from the manufacturer, two clinicians, two academics, and one representative of a user organisation. During the consensus meeting all participants brainstormed possible hazards and articulated, through discussion, the harm it could lead to. The meeting resulted in a comprehensive spreadsheet of hazards and corresponding harms. The spreadsheet was compiled and circulated for members to review, revise and populate with any further suggestions.

Hazard analysis

After identifying hazards, a hazard analysis needs to be performed. While manufacturers may be experienced in providing this for the technical side of their products, the majority of hazards for DMHIs will be related to clinical risks. Most manufacturers will be unable to assess these and will require the clinical and academic team to become conversant with applying and interpreting risk assessment procedures. To this end, the STOP clinical and academic team received training in risk assessment from a regulatory consultant who worked with them to implement the process outlined below.

The first step in a hazard analysis is to quantify the probability (likelihood) and severity (impact) of each identified hazard. First, one must assess the probability that each hazard will lead to the specified harm. One must assume that the hazard has occurred and then ask oneself “how likely is the harm to now happen?”. Some of these may be fairly standard assessments or known within the digital industry, for example, should a participant stare at the screen for longer than advised, how likely is it that they will experience physical side effects such as eyestrain, fatigue or headache? However, in many cases nuanced clinical judgements are required to make this assessment. For example, how likely is it that the participant’s condition will worsen in the short term as a result of engaging with the content of the therapy? Second, one must assess the severity and impact, should that harm occur. For example, were eye strain, fatigue or headache to occur as a result of using the device how severe could those effects be at their worst? Severity in risk assessment can be operationalised in these terms:

The duration of harmful effects.

The level of intervention or support needed in response to the effects.

The possibility and extent of any long-lasting or permanent impact.

Central to the STOP hazard analysis was the need to create bespoke probability and severity scales relevant to the clinical therapeutic context. These were carefully devised by consensus discussion between the regulatory expert consultant and members of the clinical academic team to agree the most appropriate exact thresholds and wording for both dimensions. The application of the preliminary hazard analysis for STOP was a quantitative assessment of each individual risk entry (i.e., hazard and corresponding harm) against the criteria for probability and severity outlined above. The product of these two scores yields a “risk score”. It is important to note that these risk scores were based on expert consensus estimations derived from their knowledge of the literature and field experience. In line with standard practice in the field of risk analysis, no formal validation was conducted.

Risk control and re-evaluation

In common with all risk assessments, the next stage was to work through each of the identified risks outlining all the “risk control” actions (i.e., mitigations) that could be taken to reduce the identified risk to participants as far as possible. Each risk is then re-evaluated in terms of its probability and severity yielding a revised (“post risk control”) risk score. Finally, a risk acceptability management plan is implemented where various actions are specified for any residual risks that cannot be reduced any further, for example adding “warnings” and “cautions” to device details and labelling. These serve to alert the user to important residual risks that cannot be addressed in any other way. One residual risk that is common to mental health interventions is the possibility of users being distressed when presented with information that relates to the mental health condition they are living with.

In the case of STOP, residual risks were managed using a variety of processes, depending on the nature of the risk. This included, for example, warnings (e.g. “If negative feelings or symptoms worsen as result of using this app for more than a day, please contact your support team and cease use of the STOP app until advised further”), fortnightly check-in phone calls with researchers; a dedicated, in-app 24-h study helpline number and use of an inbuilt mood-tracking algorithm to trigger researcher alerts. Further details of these are provided below.

Evaluating the safety of a DMHI within the proposed research study

After demonstrating the current safety of STOP as seen in section A of this paper, the team needed to demonstrate how the safety of STOP would be assessed in the proposed clinical trial. Any assessment of safety will involve collecting data about the occurrence of adverse events, both related and unrelated, to the trial. For STOP, we planned to do this proactively and regularly in line with recent recommendations [ 11 ]. We therefore needed an overarching framework to organise and classify the large quantity of adverse event information that was likely given the larger sample size (273) and length of time each participant would spend in the trial (6 months). We therefore devised an adverse event classification framework as follows.

AE classification framework development

Literature review

First a brief narrative literature review (conducted within the limited, 60-day time window of the regulatory approval pathway) was carried out to identify any publications in the last 10 years [03/23/2012–03/23/2022] that discussed how adverse events were assessed, coded or categorised in psychiatric populations receiving psychological interventions (digital or non-digital). We combined the categories and definitions identified in the outputs of the literature review to create a first working draft of a classification framework.

Expert consultation

We then carried out an expert consultation involving key members of the STOP trial academic team and other project stakeholders. This included the McPin Foundation, key members of the software manufacturer, a regulatory consultant and key external members of the trial committees. Full details are given in Appendix A. The classification framework working draft was shared with this group to review and comment upon. The group was invited to edit, remove or add categories or examples. Where any conflicts or differences of opinion emerged, these were resolved by group discussion and consensus using virtual meeting and/or email communications. This resulted in a finalised ‘Adverse Event Category Framework for Digital Therapeutic Devices’ which is provided in the “ Results ” section.

Proposed safety plan for the trial

To appropriately and sufficiently assess the safety of a DMHI, regulators expect to see safety positioned as the primary outcome in the proposed study, alongside efficacy. In the STOP trial, this was done by adjustment of the protocol in three ways.

First, we built-in proactive, fortnightly collection of AE data for each participant throughout the entire trial (including throughout the follow-up period) in both arms, using a custom designed checklist based upon the Adverse Events Category Framework for digital devices described above. Even though collecting AE data in both arms is resource-intensive, it is important, as shown in a recent systematic review [ 11 ]. These data enable researchers to statistically compare the prevalence of AEs in the treatment and control arms, allowing for conclusions about the safety of the DMHI. The checklist was developed from the framework, customised to the STOP trial and designed to be administered by researchers during a 10-min phone or video interview with participants. Customisation included adding introductory scripting, one or more prompt questions under each adverse event category, examples of typical events for researchers’ reference and reordering/ grouping categories and questions to optimise efficiency and acceptability of the delivery. According to the ICH-GCP guidelines, all AE data need to be categorised based on seriousness, severity, relatedness and expectedness [ 16 ]. This was done following standard guidance widely available across clinical trials units (see Appendix C for further details). The checklist was devised to incorporate the first three of these evaluations (seriousness, severity, relatedness). By definition, any event that fell within one of the listed Adverse Event Categories was considered “expected” (i.e. anticipated). Items that had not been foreseen and were therefore classed as “unexpected” were listed under the “Other” category heading. The resulting Adverse Events Checklist for the STOP trial is presented in the “ Results ” section.

In response to regulatory safety concerns, the frequency of AE data collection calls was increased to once a week for any participants identified as high risk. High-risk participants were identified at baseline using a cut-off score on a Persuadability/Suggestibility scale [ 17 ] (higher suggestibility can lead to higher risk, as the intervention aims to foster nonparanoid and trusting thoughts) and a suicide risk assessment, and throughout the trial using a suicide assessment that was administered on a weekly/biweekly basis. In addition, researchers recorded any AE that was spontaneously reported by the participant at any other contact. Note that it is crucial to collect AE data using identical methods for both the intervention and control groups even if the trial is unblinded as the control group serves as an important baseline for adverse event occurrences.

Second, we built in safety monitoring within the device. An algorithm was used to trigger an alert to researchers whenever a participant had a worsening of state mood on self-reported levels of paranoia, anxiety or sadness across a weekly treatment session (using visual analogue scale in-app pre/post session assessments; see Supplementary File 1) on 3 consecutive occasions. Researchers would then make a follow-up call to check in on the participant, collect further information and safety data and decide if follow-up action was needed (for example alerting a GP or clinical care team).

Third, we added a specific outcome measure relevant to safety, namely the Negative Effects Questionnaire (NEQ) administered once at the end of the intervention (end of treatment). The NEQ is a 20-item self-report measure [ 18 ]. It was developed using the results from Rozental et al., (2014)’s consensus statement on the negative effects of internet interventions [ 18 ], and studies aimed at investigating the negative effects of psychotherapy [ 18 , 19 ]. It is used to collect data on the negative effects experienced by patients/users during treatment, their severity and whether they were related to the intervention or other circumstances [ 19 ]. The NEQ is a reliable and valid measure with an internal consistency of α  = 0.95 [ 19 ].

Demonstrating the existing safety of the DMHI

The feasibility study main outcome paper reported no adverse events or serious adverse events and an “absence of evidence of any harmful effects on state mood and the practicality of the protocol as delivered” [ 12 ] The results from the VAS showed that there was no evidence of significant short-term detrimental effects on anxiety, sadness, paranoia or friendliness in the intervention group compared to the control group, suggesting that the intervention did not exacerbate negative mood, or pose any risk of harm to patients with distressing paranoia [ 12 ]. These data are provided in Supplementary File 1. The STOP study team used these combined findings to argue in support of the safety of STOP based on its similarity in therapeutic content to CBM-pa.

Literature search and regulatory databases

Results of literature review 1 (Device use or experience):

The search for the first literature review resulted in 14 included studies. See Appendix D for the respective PRISMA flowchart. Results showed evidence that cognitive impairment in this population does not affect engagement with digital interventions [ 20 ]. There was evidence suggesting that digital interventions are effective at improving social functioning [ 21 ], memory [ 22 ], educational and vocational attainment [ 23 ], personal recovery [ 24 ], and alleviating loneliness [ 25 ] in psychotic disorders. Some digital interventions used in this population aimed to monitor symptoms such as sleep [ 26 ], and psychotic symptoms [ 24 ]. The results of a previous literature search [ 27 ] showed that there were three digital mental health interventions that have been developed to improve symptoms in individuals struggling with psychosis [ 21 , 23 , 25 ]. The review included eight papers on smartphone-based interventions for psychosis, of which three were protocols, two were feasibility studies, two were pilot RCTs and only one was an RCT with a sample of 36 participants. This RCT found that participants who used Actissit (a Cognitive Behavioural Therapy based app for psychosis) plus treatment as usual experienced better improvements psychotic symptoms compared to those who used a symptom monitoring app plus treatment as usual [ 28 ].

The data on the use or experience of digital therapies to monitor, reduce symptoms or improve recovery in this population were promising but still limited. Larger randomised controlled trials are needed. There was no study on the use or experience of digital mental health interventions in a sample specifically defined by paranoid symptomatology except for the feasibility study precursor to the STOP [ 11 ]. For that, the literature search criteria was expanded to include devices that address psychosis in general to find comparable studies.

Results of literature review 2 (Device safety):

The search for the second literature review resulted in five included studies. See Appendix D for the respective PRISMA flowchart. Although the literature on the safety of digital mental health interventions targeting paranoia/psychosis is limited, all the current studies demonstrated positive safety outcomes [ 21 , 23 ]. A number of studies assessed the safety of the Horyzons—an online social media-based intervention that was designed to enhance social functioning in individuals with a first episode of psychosis; the studies found Horyzons safe to use (no incidents) and Horyzons users reported feeling safe and empowered [ 23 , 29 , 30 ]. A social media-based intervention called (MOMENTUM), which aims to improve social functioning in “at high-risk mental state” young individuals, was found to be safe to use [ 29 , 30 ]. A randomised controlled trial ( N  = 36) of Actissit—a CBT-informed mobile phone app for people who have experienced psychosis—found it safe to use (no serious adverse events) [ 28 ]. Finally, a randomised clinical trial ( N  = 41) assessing the EMPOWER app (Early signs Monitoring to Prevent relapse in psychosis and prOmote Wellbeing, Engagement and Recovery) reported 9 adverse events that were related to the app such as increased feelings of paranoia, increased fear of relapse and technical issues [ 31 ]. Findings were in line with those of a systematic review on the digital interventions for early psychosis where all eight smartphone-based interventions under study were found to be safe [ 27 ].

The clinical data appraisal tools for both literature reviews are provided in Appendix C.

Regulatory databases

There were no safety concerns raised from the review of the regulatory databases.

In total 26 unique hazards and their corresponding harms were identified, which are listed in full in Appendix E.

The team defined likelihood/probability and severity for the proposed study as explained in the “ Methods ” section. Table 1 shows the final operationalised definition of probability and Table  2 shows the equivalent for severity.

Afterwards, the team used these definitions to rate the probability and severity of every identified hazard. Probability refers to the likelihood that the identified hazardous situation will lead to the specified harm . A risk score was calculated for each hazard entry by multiplying the probability and severity scores.

Under this step, the study team identified all measures they could take to reduce each risk entry as much as possible, listing these as “risk controls”. They then recalculated new probability, severity and risk scores under the assumption that the stated risk controls were effective. Before implementing risk-control strategies, the highest risk score was 20 out of 36. After applying these strategies, the highest risk score was reduced to 9 out of 36. These quantitative ratings and products are shown in Appendix F, which constitutes the final STOP Hazard Analysis, and was a key requirement of the submission for regulatory approval. One new insight that emerged from the consultation was the need for a product recall feature that could operate at either an individual or entire cohort level. This requirement was a crucial safety attribute, for use in the unlikely event that access to the app had to be immediately terminated. At the individual level access could be revoked by account deactivation. At the cohort level, the technical team implemented a “recall switch” feature in the app to enable the recall, and a corresponding participant facing message. Footnote 1

The literature review and the expert consultation that we conducted as described in the “ Methods ” section resulted in a finalised “Adverse Event Category Framework for Digital Therapeutic Devices” (See Table  3 ) . This framework was informed by three key publications arising from the literature review that discussed a range of classes of negative effects in psychotherapy [ 18 , 32 , 33 ]. Our framework was then used to identify the “anticipated” AEs for the STOP trial. As such, it might be applicable to all DMHIs that are in the form of a mobile app. The AEs collected are not exclusive to STOP but might not all be relevant or be comprehensive of all possible AEs for another DMHI. Professionals testing a DMHI delivered in a different format (virtual reality for example) and/or targeting a different population would need to make suitable adjustments and might even wish to incorporate additional AE categories specific to their device’s safety profile. However, this framework is recommended as a potentially useful starting point for any DMHI.

Some of the categories such as technical malfunction might be less clear to clinicians, as they do not directly relate to the therapeutic component of the device. When assessing adverse events (AEs), it is essential to evaluate the entire device, not just the treatment component. This includes potential risks from using any mobile app. Furthermore, the MHRA approval mandates monitoring all aspects of the approved research for safety, covering study procedures, intervention and the device. One learning that came out of discussions during the development of the AE framework with other professionals was the need to have a separate AE category for “device deficiency” that is distinct from “technical malfunction”. Device deficiency is defined as “an inadequacy of a medical device related to its identity, quality, durability, reliability, safety or performance, such as malfunction, misuse or use error and inadequate labelling” [ 34 ]. This differentiation was highlighted by some of the academics on the team with experience in other DMHIs, to align with the requirements and terminology used by the regulatory framework.

The trial is still ongoing at the time of writing and a full report of the STOP safety evaluation will be published as part of the trial outcomes. Here we present the tools we developed to aid STOP safety data collection, as described in the “ Methods ” sections of this paper. We also outline the final STOP safety analysis plan, which was subject to rigorous review and revision as part of the regulatory approval process.

Safety data collection

The Adverse Events Checklist used by researchers to proactively collect fortnightly (or weekly for more vulnerable participants) AE data is presented in Appendix G. Participants were asked each prompt question in turn to identify and record details of any adverse that had happened since the last researcher contact. For every event recorded researchers completed the remaining columns of the checklist to record a free text event description and to determine its seriousness, relatedness, expectedness and severity. The checklist will be administered every week rather than fortnightly with high-risk patients to mitigate any risks. It is likely that administering the checklist in these patients more than the rest might lead to a higher number of reported AEs. This will be taken into account in the analyses using sensitivity analyses.

Safety data analysis

The complete statistical analysis plan (SAP) for the STOP trial underwent a number of iterations in review with the regulators before approval was achieved. In terms of safety specifically, the approved plan included the following. Any AE/SAE involving the target clinical symptoms (paranoia) will be analysed separately from other AE/SAEs, due to the assessed (small) likelihood that the device could trigger these symptoms. This risk was singled out for separate analysis because it was the one of most concern to clinicians and regulators. Formal statistical analyses are unlikely due to small numbers of observations but the incidence rate of AEs (total number of those having the event divided by the person-months at risk) and the ratio of incidence rates of AEs between the two treatment arms per time period will be reported to allow detection of any safety concerns within the treatment arm.

Analysis of the checklist data will produce a list of adverse events along with frequencies, seriousness, relatedness and possible methods of prevention/mitigation. Additionally, demographic and clinical characteristics of those who experienced adverse events will identify patients who might be at a higher risk. Comparative statistical tests will be used to analyse the NEQ data between the treatment and control arms using a linear regression approach.

An overview of the pathway followed by the STOP team from start to finish is provided in Table  4 . This shows the purpose of each step, some brief details on what it included and pointers allowing the reader to more easily navigate to relevant sections of the present paper and associated resources.

This paper details the steps that the STOP study team took to thoroughly assess the safety of a DMHI and achieve regulatory approval to conduct an RCT (MHRA’s NoNO). The example shared in this paper serves as a guide for academics and other professionals in the field. It provides a roadmap for the essential prerequisites, requirements and expectations regarding safety when seeking regulatory approval to conduct research with DMHIs. A fuller understanding of this pathway will significantly benefit research teams, clinicians and developers involved in the process of developing and delivering novel DMHIs.

There are various key concepts and practical takeaways outlined in this paper. The overarching requirement is to compile an evidence-based argument that the benefit of the proposed device outweighs its risk to users, and this can only be done convincingly by the fullest consideration and quantification of that risk. The process by which one might do this can be broken down into various discrete steps. Figure  1 demonstrates the process model presented in this paper.

The process model of “How to demonstrate the safety of as-yet untested DMHI?

First, it is important to establish the safety of the DMHI even before testing its efficacy. This could be done by looking at the safety data of “equivalent” interventions that have been used in a similar population, studying the literature and/or conducting a feasibility/pilot study to assess the preliminary safety of the intervention. It is noteworthy that in the UK devices exclusively developed and used (either clinically or for research) within a single institution are exempt from formal regulatory approval requirements [ 35 ] which can provide an appropriate setting for gathering early-stage safety data. Second, conducting a comprehensive risk analysis specific to each DMHI is crucial [ 36 ]. This involves identifying all the potential hazards that are relevant to that DMHI, assessing any potential harm (likelihood and severity), calculating a risk rating per identified hazard, implementing risk control measures, reassessing risk, calculating a final post-risk score, denoting and reporting any residual risk and finally demonstrating that the expected benefits outweigh the identified risks in a quantifiable manner. Third, the safety of a new and untested DMHI needs to be evaluated as a primary outcome within the proposed research. It needs to hold the same importance as efficacy/effectiveness, irrespective of the academic research agenda. A safety evaluation plan needs to be integrated within the study protocol or presented separately as a standalone study.

Fourth, a helpful component of any safety evaluation is the use of a framework for organising the data to be collected, given the likely breadth of possible adverse events. The Adverse Event Category Framework for Digital Therapeutic Devices provides one such possibility. At a more practical level, this must be supplemented by a structured approach to collecting and evaluating individual adverse events. The Adverse Events Checklist (provided in appendices) received regulatory approval for use in the STOP trial and could usefully serve as a guide for others. By incorporating categorisation of each entry on the key dimensions of seriousness, severity, relatedness and expectedness, it allows a research team to more easily demonstrate their intended compliance with reporting requirements. It also facilitates gathering a richer dataset around negative effects that will go on to permit a more comprehensive analysis than previous traditional practices [ 11 ]. A scoping review on the recording of harms in RCTs of behaviour change interventions has mapped out the categories of harms found in that literature [ 35 ]. As might be expected, there is some overlap with our AE category framework, such as physical and psychological harms, which is reassuring and validatory. In contrast, group-level harms (such as the impact of a behaviour change intervention on culture, environment or health equity) feature strongly in the scoping review but are absent from our framework, which focused exclusively on individual participant-level harms data. It will be important for future studies to consider whether macro-level harms relevant to behaviour change interventions might also be relevant to DMHI interventions.

It is important to highlight the time involved in the processes summarised in this paper. In the present worked example acquiring regulatory approval (MHRA NoNO) took approximately 6 months and the authors’ recommendation is to allow a timeframe of up to 9 months, if working from a position of relatively little prior knowledge and experience. This timeline is necessary to allow for the involvement of clinical and technical experts, patient groups and regulatory consultants. In the present worked example employing a regulatory consultant played a vital role in ensuring compliance with all regulatory requirements and smooth passage through regulatory review. Their knowledge of the complex regulatory landscape provided a key interface between software developers and the academic team to ensure that the requisite information was compiled and presented in a manner compliant with the appropriate national and international standards [ 37 ]. Academic teams are advised to routinely cost such expertise into research projects involving medical devices, unless equivalent institutional support is already available.

Limitations

It is important to be aware that this paper provides an example of how one DMHI assessed safety and achieved regulatory approval. The experiences of other DMHIs will most likely differ. Thus, it is important to view this process flexibly and adapt it to each DMHI. Furthermore, this example is UK-centric. Even though the process described might be helpful for DMHIs applying for regulatory approval outside the UK, professionals need to be aware of the needs of their specific regulatory environment.

The example provided in this paper can be adapted by other professionals in the digital mental health field to help them navigate complex regulatory processes. Prioritising and emphasising safety and regulatory compliance allows researchers to contribute to the responsible development of DMHIs. Ensuring that the benefit of these interventions outweighs any risks that they carry is important for building confidence and trust among clinicians, patients and academics. The systematic approach to safety evaluation outlined here sets a valuable precedent for assessing the safety of DMHIs.

Availability of data and materials

All data generated or analysed during this study are included in this published article in the form of tables and appendices.

“ You can no longer use STOP as the product has been withdrawn. You will shortly be contacted by a member of the research team who will explain and offer further support if required. In the meantime, if you require more urgent assistance, please contact the study helpline at 020 784 80,425 or clinical support email at [email protected]”.

Abbreviations

Adverse event

Digital mental health intervention

International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use—Good Clinical Practice

Medicines and Healthcare products Regulatory Agency

Negative Effects Questionnaire

National Health Service

National Institute of Health and Care Excellence

• Notification of No Objection

Preferred Reporting Items for Systematic Reviews and Meta-Analyses RCT: randomised controlled trial

Randomised controlled trial

Serious adverse event

Software as a Medical Device

Philippe TJ, Sikder N, Jackson A, et al. Digital health interventions for delivery of mental health care: systematic and comprehensive meta-review. JMIR Ment Health. 2022;9(5):e35159. https://doi.org/10.2196/35159 .

Article   PubMed   PubMed Central   Google Scholar  

Sanderson C, Kouzoupi N, Hall CL. Technology matters: the human touch in a digital age – a blended approach in mental healthcare delivery with children and young people. Child Adolesc Ment Health. 2020;25(2):120–2. https://doi.org/10.1111/camh.12385 .

Article   PubMed   Google Scholar  

Gómez Bergin AD, Valentine AZ, Rennick Egglestone S, Slade M, Hollis C, Hall CL. How are adverse events identified and categorised in trials of digital mental health interventions? A narrative scoping review of trials in the ISRCTN registry (Preprint). JMIR Ment Health. https://doi.org/10.2196/42501 . Published online February 22, 2022.

National Institute for Health and Care Excellence (NICE). Evidence standards framework for digital health technologies. https://www.nice.org.uk/corporate/ecd7/resources/evidence-standards-framework-for-digital-health-technologies-pdf-1124017457605 .

MHRA. Notify the MHRA about a clinical investigation for a medical device. https://www.gov.uk/guidance/notify-mhra-about-a-clinical-investigation-for-a-medical-device .

MHRA. Clinical investigations of medical devices – compiling a submission to MHRA. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1097797/Guidance_for_mfrs_-_compiling_a_submission_to_MHRA_-_May_2021.pdf .

Guideline IH. Clinical safety data management: definitions and standards for expedited reporting E2A. In: International conference on harmonisation of technical requirements for registration of pharmaceuticals for human use 1994 Oct. 1994.

Google Scholar  

Papaioannou D, Cooper C, Mooney C, Glover R, Coates E. Adverse event recording failed to reflect potential harms: a review of trial protocols of behavioral, lifestyle and psychological therapy interventions. J Clin Epidemiol. 2021;136:64–76. https://doi.org/10.1016/j.jclinepi.2021.03.002 .

Papaioannou D, Sprange K, Hamer-Kiwacz S, Mooney C, Moody G, Cooper C. Recording harms in randomised controlled trials of behaviour change interventions: a qualitative study of UK clinical trials units and NIHR trial investigators. Trials. 2024;25(1):163. https://doi.org/10.1186/s13063-024-07978-1 .

Papaioannou D, Hamer-Kiwacz S, Mooney C, et al. Recording harms in randomized controlled trials of behavior change interventions: a scoping review and map of the evidence. J Clin Epidemiol. 2024;169:111275. https://doi.org/10.1016/j.jclinepi.2024.111275 .

Taher R, Hsu CW, Hampshire C, et al. The safety of digital mental health interventions: systematic review and recommendations. JMIR Ment Health. 2023;10:e47433. https://doi.org/10.2196/47433 .

Yiend J, Lam CLM, Schmidt N, et al. Cognitive bias modification for paranoia (CBM-pa): a randomised controlled feasibility study in patients with distressing paranoid beliefs. Psychol Med. 2023;53(10):4614–26. https://doi.org/10.1017/S0033291722001520 .

Hsu C, Stahl D, Mouchlianitis E, Peters E, Vamvakas G, Keppens J, Watson M, Schmidt N, Jacobsen P, McGuire P, Shergill S, Kabir T, Hirani T, Yang Z, Yiend J. User-Centered Development of STOP (Successful Treatment for Paranoia): Material Development and Usability Testing for a Digital Therapeutic for Paranoia. JMIR Hum Factors. 2023;10:e45453. https://humanfactors.jmir.org/2023/1/e45453 . https://doi.org/10.2196/45453 .

European Commission. Clinical evaluation: a guide for manufacturers and notified bodies under directives 93/42/EEC and 90/385/EEC. 2016.

ISO 14971: Medical devices Application of risk management to medical devices. https://www.iso.org/standard/72704.html .

International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH). Integrated addendum to ICH E6(R1): guideline for good clinical practice E6(R2). https://database.ich.org/sites/default/files/E6_R2_Addendum.pdf .

Kotov RI, Bellman SB, Watson DB. Multidimensional Iowa Suggestibility Scale (MISS) Brief Manual. 2004.

Rozental A, Andersson G, Boettcher J, et al. Consensus statement on defining and measuring negative effects of internet interventions. Internet Interv. 2014;1(1):12–9. https://doi.org/10.1016/j.invent.2014.02.001 .

Article   Google Scholar  

Rozental A, Kottorp A, Forsström D, et al. The negative effects questionnaire: psychometric properties of an instrument for assessing negative effects in psychological treatments. Behav Cogn Psychother. 2019;47(5):559–72. https://doi.org/10.1017/S1352465819000018 .

Bell IH, Lim MH, Rossell SL, Thomas N. Ecological momentary assessment and intervention in the treatment of psychotic disorders: a systematic review. Psychiatr Serv. 2017;68(11):1172–81. https://doi.org/10.1176/appi.ps.201600523 .

Biagianti B, Schlosser D, Nahum M, Woolley J, Vinogradov S. Creating Live Interactions to Mitigate Barriers (CLIMB): a mobile intervention to improve social functioning in people with chronic psychotic disorders. JMIR Ment Health. 2016;3(4):e52. https://doi.org/10.2196/mental.6671 .

Reeder C, Huddy V, Cella M, et al. A new generation computerised metacognitive cognitive remediation programme for schizophrenia (CIRCuiTS): a randomised controlled trial. Psychol Med. 2017;47(15):2720–30. https://doi.org/10.1017/S0033291717001234 .

Article   PubMed   PubMed Central   CAS   Google Scholar  

Alvarez-Jimenez M, Koval P, Schmaal L, et al. The Horyzons project: a randomized controlled trial of a novel online social therapy to maintain treatment effects from specialist first-episode psychosis services. World Psychiatry. 2021;20(2):233–43. https://doi.org/10.1002/wps.20858 .

Lewis S, Ainsworth J, Sanders C, et al. Smartphone-enhanced symptom management in psychosis: open, randomized controlled trial. J Med Internet Res. 2020;22(8):e17019. https://doi.org/10.2196/17019 .

Lim MH, Gleeson JFM, Rodebaugh TL, et al. A pilot digital intervention targeting loneliness in young people with psychosis. Soc Psychiatry Psychiatr Epidemiol. 2020;55(7):877–89. https://doi.org/10.1007/s00127-019-01681-2 .

Meyer N, Joyce DW, Karr C, et al. The temporal dynamics of sleep disturbance and psychopathology in psychosis: a digital sampling study. Psychol Med. 2022;52(13):2741–50. https://doi.org/10.1017/S0033291720004857 .

Rus-Calafell M, Schneider S. Are we there yet?!—a literature review of recent digital technology advances for the treatment of early psychosis. Mhealth. 2020;6:3–3. https://doi.org/10.21037/mhealth.2019.09.14 .

Bucci S, Barrowclough C, Ainsworth J, et al. Actissist: proof-of-concept trial of a theory-driven digital intervention for psychosis. Schizophr Bull. 2018;44(5):1070–80. https://doi.org/10.1093/schbul/sby032 .

Alvarez-Jimenez M, Gleeson J, Bendall S, et al. SU125. Momentum: a novel online social media, mindfulness, and strengths-based intervention to promote functional recovery in ultra high risk (UHR) patients. Schizophr Bull. 2017;43(suppl_1):S206–7. https://doi.org/10.1093/schbul/sbx024.121 .

Article   PubMed Central   Google Scholar  

Alvarez-Jimenez M, Gleeson JF, Bendall S, et al. Enhancing social functioning in young people at Ultra High Risk (UHR) for psychosis: a pilot study of a novel strengths and mindfulness-based online social therapy. Schizophr Res. 2018;202:369–77. https://doi.org/10.1016/j.schres.2018.07.022 .

Article   PubMed   CAS   Google Scholar  

Bradstreet S, Allan S, Gumley A. Adverse event monitoring in mHealth for psychosis interventions provides an important opportunity for learning. J Ment Health. 2019;28(5):461–6. https://doi.org/10.1080/09638237.2019.1630727 .

Linden M. How to define, find and classify side effects in psychotherapy: from unwanted events to adverse treatment reactions. Clin Psychol Psychother. 2013;20(4):286–96. https://doi.org/10.1002/cpp.1765 .

Gullickson KM, Hadjistavropoulos HD, Dear BF, Titov N. Negative effects associated with internet-delivered cognitive behaviour therapy: an analysis of client emails. Internet Interv. 2019;18. https://doi.org/10.1016/j.invent.2019.100278 .

UK Goverment. Glossary. https://assets.publishing.service.gov.uk/media/6143184cd3bf7f05b694d6cc/Glossary_PDF.pdf .

MHRA. In-house manufacture of medical devices in Great Britain. https://www.gov.uk/government/publications/in-house-manufacture-of-medical-devices/in-house-manufacture-of-medical-devices .

Khinvasara T, Ness S, Tzenios N. Risk management in medical device industry. J Eng Res Rep. 2023;25(8):130–40. https://doi.org/10.9734/jerr/2023/v25i8965 .

Medical Device Regulation. MEDDEV Guidance List. https://www.medical-device-regulation.eu/meddev-guidance-list-download/ .

Download references

Acknowledgements

We acknowledge the contributions made to this work by the McPin Foundation. We thank Andrew Gumley, Alex Kenny, Caroline Murphy, Sumiti Saharan, Carolina Sportelli and Chris Taylor for their input to consultations carried out as part of the work reported here.

This work was supported by the Medical Research Council Biomedical Catalyst: Developmental Pathway Funding Scheme (DPFS), MRC Reference: MR/V027484/1. We would also like to express our gratitude to the National Institute for Health and Care Research (NIHR) Biomedical Research Centre hosted at South London and Maudsley NHS Foundation Trust in partnership with King’s College London. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, the Department of Health and Social Care, the ESRC or King’s College London. For the purposes of open access, the author has applied a Creative Commons Attribution (CC, BY) licence to any Accepted Author Manuscript version arising from this submission.

Author information

Authors and affiliations.

Department of Psychosis Studies, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, UK

Rayan Taher & Jenny Yiend

Institute of Mental Health, School of Medicine, National Institute for Health and Care Research MindTech MedTech Co-Operative, University of Nottingham, Nottingham, UK

Charlotte L. Hall & Aislinn D Gomez Bergin

School of Computer Science, University of Nottingham, Nottingham, UK

Aislinn D Gomez Bergin

Avegen Health, London, UK

Neha Gupta & Nayan Kalnad

London Institute for Healthcare Engineering, King’s College London, London, UK

Clare Heaysman

Department of Psychology, Addiction and Mental Health Group, Bath Centre for Mindfulness and Compassion, University of Bath, Bath, UK

Pamela Jacobsen

McPin Foundation, London, UK

Thomas Kabir

Department of Informatics, King’s College London, London, UK

Jeroen Keppens

Department of Psychological Medicine, Dunedin School of Medicine, University of Otago, Otago, New Zealand

Che-Wei Hsu

Department of Psychiatry, University of Oxford, Oxford, UK

Philip McGuire

Department of Psychology, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, UK

Emmanuelle Peters

Kent and Medway Medical School, Canterbury, Kent, UK

Sukhi Shergill

Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, UK

Daniel Stahl

Wensley Stock Ltd, Wiltshire, UK

Ben Wensley Stock

You can also search for this author in PubMed   Google Scholar

Contributions

RT, CH and JY conducted the literature reviews. CWH, TK, CH, PM, EP, SS, BW, DS and JY were involved in the development of the risk management plan. RT, CWH, TK, CH, PM, EP, SS, BW, DS and JY were involved in the development of the Adverse Events Checklist. RT, AB, CH and JY wrote up this paper. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Jenny Yiend .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Taher, R., Hall, C.L., Bergin, A.D.G. et al. Developing a process for assessing the safety of a digital mental health intervention and gaining regulatory approval: a case study and academic’s guide. Trials 25 , 604 (2024). https://doi.org/10.1186/s13063-024-08421-1

Download citation

Received : 30 November 2023

Accepted : 21 August 2024

Published : 10 September 2024

DOI : https://doi.org/10.1186/s13063-024-08421-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Regulatory approval
• Digital mental health
• Digital mental health interventions
• Adverse events
• Medical device
• Software as a medical device

ISSN: 1745-6215

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Healthc Eng
• v.2023; 2023
• PMC10241585

Strategies for Medical Device Development: User and Stakeholder Perceptions

I-ching tsai.

1 Department of Biomedical Engineering, National Cheng Kung University, No. 1, University Road, Tainan City 701, Taiwan

Ching-Da Wang

Peng-ting chen.

2 International Institute of Medical Device Innovation, National Cheng Kung University, No. 1, University Road, Tainan City 701, Taiwan

Associated Data

The data used in this article cannot be shared publicly due to privacy reasons of the participants of the study.

Medical device development involves user safety, and it is governed by specific regulations. The failure of medical device developers to consider the influence of users, the environment, and related organizations on product development during the design and development process can result in added risks to the use of medical technologies. Although many studies have examined the medical device development process, there has been no systematic and comprehensive assessment of the key factors affecting medical device development. This research synthesized the value of medical device industry stakeholders' experiences through a literature review and interviews with industry experts. Then, it establishes an FIA-NRM model to identify the key factors affecting medical device development and suggests appropriate pathways for improvement. Results indicate that the development of medical devices should begin with stabilizing organizational characteristics, followed by strengthening technical capability and use environment, and finally, consideration should be given to the user action of medical devices. The results provide medical device developers with optimal development pathways and resource allocation recommendations to support developers in developing medical device development strategies as well as ensuring the safety and effectiveness of the products for end users.

1. Introduction

Medical devices have brought numerous benefits and contributions to human health; however, regulations and medical particularities increase the costs and sensitivity of medical device research, design, and clinical applications [ 1 ]. These situations pose considerable challenges to the developers, especially small-to-medium enterprises (SMEs) with limited operational resources [ 2 , 3 ]. When the medical device industry fails to evaluate numerous medical device designs for the usability in the product development process, it can lead to various problems in their applicability and can give rise to risks owing to the insufficient consideration of human, environmental, and organizational factors [ 4 , 5 ]. Therefore, medical device developers need to be aware of the various aspects of the development process and focus on human factors and compliance with medical device marketing regulations to reduce postmarket risks. However, an important research gap exists regarding how to invest limited resources in product development and how to formulate development strategies to achieve optimal healthcare benefits based on product efficiency, medical regulations, and user needs [ 6 ].

Many studies have examined various key factors in the medical device development process, including user operation [ 7 ], risks [ 8 ], effectiveness and regulation [ 9 ], and public stereotypes [ 10 , 11 ]. Although research in medical device development continues to grow in the field of medical engineering, there is still a lack of a more systematic and comprehensive research framework based on stakeholder perspectives to evaluate the crucial elements of the medical device development process [ 12 ]. There is a need for empirical accounts of medical device development factors as perceived by stakeholders. This study analyzed the key factors in the medical device development process using stakeholder interviews and questionnaires. The results from this study can help medical device developers to prioritize and allocate resources to critical items in the medical device development process.

2. Literature Review

2.1. medical device innovation challenge.

The growing demand for healthcare has made medical devices increasingly important in the healthcare industry. Countries classify medical devices according to their associated “user risk” as a basis for quality and safety management, regulatory control, or market licensing. Owing to the particularity of the medical industry and devices, governments around the world have clear classifications and strict specifications for the development and marketing of medical devices. Such regulations make the design, development, and market planning of this industry more challenging than those of general commodities [ 6 ]. One of the problems of medical device usage involves the medical device developer's failure to consider fully the application status of a product during the product design and development process, thereby resulting in equipment inapplicability. This problem may be due to a mismatch between user characteristics, product interface, and functions or the usage environment, which may affect users' cognition or cause operation errors, thereby leading to risks and injuries [ 13 ]. Overall, the ability to effectively integrate input from developers, organizations, and users in the early stages of medical device development to evaluate product design, confirm usability and save costs to develop safe and effective medical device products will provide sustainable benefits to developers and users.

2.2. Medical Device Development Considerations

The main purpose of a medical device is to meet indications for use and user needs. Medical device developers are often unable to understand the benefits of focusing on human engineering in the development of medical devices due to their inability to implement user involvement in the design process [ 14 ]. Therefore, developers must provide product education and training to improve operation skills and increase user confidence and trust in products [ 15 ]. In addition to incorporating human behaviour, capabilities, and limitations into the design of medical product systems, it is crucial to take into account individual user differences [ 16 ]. This includes technical knowledge, experience, and education.

Medical devices can be used in clinical or nonclinical settings, such as community homes and public settings [ 17 ]. Numerous factors related to the environment, organizational characteristics, and user status can affect the use of a medical device [ 18 ]. In the development process, the characteristics of an intended use environment (e.g., time, pressure, lighting, noise, temperature, and physical layout) can help developers understand the operation of a product and can optimize the use efficiency [ 19 , 20 ] as well as improve safety and effectiveness. The development of medical devices requires consideration of the opinions of different stakeholders, the management, and the culture that can be critical to the success of the product. A variety of factors can influence the product development strategy of medical device developers, including cost control, professional manpower, budget availability, and performance expectations [ 21 ]. Given the particularity of medical devices, market strategies should consider social backgrounds, reimbursement processes, and regulatory policies [ 9 ].

This study identifies four dimensions of medical device development, including user action (UA), technical capability (TC), use environment (UE), and organizational characteristic (OC), to comprehensively assess the key factors of medical device development as a piece of advice for medical device development and medical industry.

3. Research Design

After conducting a literature review and examining case studies of medical device development, this study gathered specific information that influenced medical device development. For the professional experience data collection, semistructured interviews with key stakeholders were used. Key stakeholders include two managers of auditing organisations, a CEO of a consultant company, two managers of a government agency, and a product manager of a medical device company. The interview included open-ended questions, focusing on the process of medical device development and key factors. From the interviews, key aspects and factors related to medical device development were extracted and used to design questionnaires. The questionnaires were distributed among experts and their responses were recorded. The DEMATEL method was employed to analyze the questionnaire data, resulting in the establishment of a FIA-Network Relationship Map (NRM) model. The outcomes of this analysis demonstrate the importance and interaction degree for each key factor. Finally, based on the results of the FIA and NRM, suggestions were proposed for the development of the medical device development process ( Figure 1 ).

Research framework flowchart.

3.1. Participants

The questionnaire design was based on a literature review and content analysis of expert interviews. The questionnaire investigates the background of the respondents, including their age, work experience, professional knowledge, and the organization they work for, to confirm that the respondents conform to the current study on the medical device development process. After the questionnaire was distributed, a total of 65 valid samples were collected. They work in different professions related to health care and medical equipment. Among the medical device developers that we surveyed, 23 (36.0%) were from medical profession background, 32 (49%) were engineering expertises (including technical staff and managers), and 10 (15%) were from regulatory expertises. The respondents included 48 males (74%) and 17 females (26%). The average of seniority distribution of the respondents in the biomedical industry was 9.1 years, 78% are above 5 years, and 40% are above 10 years. The study also examined the level of medical device risk involved in the development experience of the respondents, including 5 (8%) who were involved in Class I risk, 33 (50%) who were involved in Class II risk, and 27 (42%) who were involved in Class III risk.

3.2. Content Analysis

The content analysis uses qualitative or quantitative data and involves methods of induction or deduction. Content analysis, which is also known as text or literature analysis, converts qualitative data into quantitative data for analysis. The value of content analysis lies in its utilization of system objective and quantitative methods to classify statistics. The hidden content of records can be systematically organized and visualized based on the narrative interpretation of the numbers in the categories. Content analysis methods are applicable when sorted verbal information is critical to the research [ 22 , 23 ].

Content analysis is an objective and systematic method for investigating and analyzing the content of documents and clearly describing the content of the communication. Moreover, it can analyze various languages and features in communication content [ 24 ]. The possibility of exploring a particular property of information can assist the prompt deduction of meaning. Additionally, the content analysis examines and analyzes communications to measure variables quantitatively, objectively, and systematically. Beneficial and simple, content analysis has been used in numerous aspects of scientific research for over six decades. The hypothesis of the present study states that the most frequently mentioned words reflect the biggest problem. Content analysis involves three steps, namely, unit coding, sampling, and validity analysis [ 25 , 26 ].

The experts interviewed in this study were experienced in the development, management, and use of medical devices and provided valuable advice during the interviews. Content analysis was used to analyze the interviews.

3.3. DEMATEL Method

The DEMATEL method can be used to study and solve complex and interwoven problem sets [ 27 ]. The DEMATEL method enabled the researchers to understand specific problems and interweave clusters of problems as well as to better identify possible solutions through hierarchical structures. Recent studies have used DEMATEL techniques to solve complex problems, such as the analysis of smart product service systems [ 28 ], probabilistic safety analysis of process systems [ 29 ], pharmaceutical manufacturing [ 30 ], and hospital performance management [ 31 ]. This method differs from traditional methods in that an NRM can identify interdependence among system elements through causal graphs. In this research, the DEMATEL method was applied to constitute NRMs to investigate whether the development processes of medical device design interact with one another or they are independent. The concept of the DEMATEL method is as follows [ 27 ]. Calculate the average matrix: first, organize actors through the questionnaire and obtain interactions among the factors. Each respondent will be asked to assess the direct impact of any two factors with an integer score ranging from 0 to 4, (0 = “no influence,” 1 = “low influence,” 2 = “medium influence,” 3 = “high influence,” and 4 = “extreme strong influence”). The next step is to establish the initial influence matrix: The impact between two pairs of factors will be compared in the survey questionnaire. X ij indicates the extent to which a respondent considered factor i affecting factor j , and the diagonal of the matrix shows the influence of the factor on itself, it will be set to 0 when there is no influence.

The next step is to establish the normalized direct-influence matrix: A normalized datum is the maximum of row vectors and the sum of column vectors. The normalization influence matrix is denoted by M , and the normalized datum is set to s . M and s can be calculated as follows:

To calculate the indirect-influence matrix, the indirect-influence matrix is set to IM. The indirect-influence matrix can be gained by directly affecting the value of matrix ( M ) calculated by the following equation:

To calculate the total-influence matrix, the value of the total-influence matrix can be obtained from the value of the direct-influence matrix, and the value of the indirect-influence matrix can be calculated using the following equations:

Subsequently, the structural relationship between the factors is analyzed.

The sum vector of the row value is d i , and the sum vector of the column value is r i . Then, if we let i  =  j , the sum vector of the row value plus the column value will be ( d i + r i ), which represents the center degree. If the sum of the row value plus the column value ( d i + r i ) is high; thus, the relationship among dimensions or criteria will be powerful. The sum of the row value minus the column value is ( d i − r i ), which indicates the extent of the reason. If d i − r i  > 0, then the degree of influence on others is stronger than the degree of being influenced; otherwise, d i − r i  < 0. Finally, the center degree ( d i + r i ) is taken as the X axis and the reason degree ( d i − r i ) is taken as the Y axis.

In this study, the structure influence relation diagram was drawn. Next, the relation diagram was divided into four quadrants by the average of the center and reason degrees. The distributions of the indices were observed on the influence network diagram, and the causality and core degree of the index were analyzed.

3.4. FIA Model

Martilla et al. originally proposed the importance-performances (IPA) model to verify the importance and performance of factors being investigated, thereby dividing the two axes into four quadrants and indices [ 32 ]. Based on this model, decision-makers can sort through and improve the relevant attributes of their products or services. The IPA model does not waste resources on inappropriate and informal strategies and has long been considered a simple and effective technique. The present study extended its analysis of the FI and II. As shown in Figure 2 , four frequency quadrants were constructed with frequency indicators based on the weighted survey provided by the respondents, and the impact indicates decision-makers to make strategic decisions. This study proposed four service improvement strategies for analyzing the four frequency and impact indicators.

•   Priority: The first quadrant illustrates a high level of frequency and impact (H, H). This quadrant demonstrates that a factor has a high frequency and a high impact. Thus, medical device companies can prioritize solving this factor to strengthen their design development. In this research, we named this quadrant “Priority.”
•   Investing resource: The second quadrant illustrates a low level of frequency and a high level of impact (L, H). The quadrant demonstrates that a factor has a high impact but does not reflect frequency. Therefore, medical device companies should invest resources in response to this factor. In this research, we named this quadrant “Investing resource.”
•   Standstill: The third quadrant illustrates a low level of frequency and a low level of impact (L, L). This quadrant shows that factors situated in it have a low frequency and a low impact; thus, medical device companies can maintain their current status. In this research, we named this quadrant “Standstill.”
•   Suspension: The fourth quadrant illustrates a high level of frequency and a low level of impact (H, L). This quadrant shows that the impact is not large, but the frequency is high. Medical device companies can suspend processing first. In this research, we named this quadrant “Suspension.”

3.5. NRM Analysis

The purpose of the DEMATEL method is to form a network diagram (i.e., an NRM). In addition, the method is mainly used to determine if factors interact or are independent, and the NRM is the final step in the DEMATEL method. The relationship between the degree and level of interaction of factors can be described by an easy-to-understand structure and a precise simplification of interdependence [ 33 ]. The NRM differs from the FIA model in the sense that it assigns and ranks factors based on specific characteristics. The NRM reveals the interrelationships among factors and evidence that provides additional important factors. A structural matrix and causal map can be used to show causality and impact, and the factors in a complex system can promote decision-making [ 34 , 35 ].

4. Data Analysis and Results

4.1. content analysis.

Based on the literature review, case studies and interviews with experts, the process of medical device development is divided into user action (UA), technical capability (TC), use environment (UE), and organizational characteristics (OC). To answer our research question (what are the challenges in the development of medical devices?), we interviewed experts with experience in medical device development in Taiwan. These experts come from a variety of medical device-related organizational backgrounds, including audit organizations, consulting firms, government agencies, and medical device companies. For this study, a minimum of six years of experience in the development or evaluation of multiple medical devices is required for someone to be called an expert. The respondents were interviewed face-to-face with informed consent in order to understand the key factors in the development of current medical devices. An overview of the stakeholders is provided in Table 1 . Three experienced coders are responsible for the verbatim coding of the interviews for content analysis. The UA, TC, UE, and OC profiles and 16 key factors were extracted from the interviews and cited in the literature or the expert interviews. In addition, these key factors were clearly defined and analyzed for reliability ( Table 2 ). Three coders performed the coding. Coders have experience in medical device innovation and underwent several rounds of practice coding with subsamples. Coders calculate the number of factors that each coder overlapped and then calculate mutual agreement and reliability. Disagreements were resolved after discussions and reassessments of the case to eventually arrive at a consensus. Table 3 shows that the average mutual agreement between the coders is 0.844, which is high. In addition, the reliability test presents that the reliability of the three coders interviewed is 0.942, and the values greater than 0.8 represent the high reliability of content analysis [ 36 ].

The interviewee's background information.

Assessment structure of medical device development.

Note: A: [ 14 ], B: [ 15 ], C: [ 16 ], D: [ 37 ], E: [ 38 ], F: [ 39 ], G: [ 8 ], H: [ 40 ], I: [ 41 ], J: [ 42 ], K: [ 17 ], L: [ 18 ], M: [ 43 ], N: [ 44 ], O: [ 19 ], P: [ 20 ], Q: [ 45 ], R: [ 46 ], S: [ 47 ], T: [ 21 ], U: [ 9 ].

Reliability and mutual agreement between coders.

0.844 indicates high agreement between coders for content analysis. 0.942 is greater than 0.8 indicating high reliability of content analysis.

4.2. FIA-NRM Model

The purpose of FIA is to conduct placement positioning based on the dimensions and the factors FI and II such that the medical device developers can have an adequate command of the frequency and impact of each dimension or factor. To build the FIA models, the average values of the weights (from 0 to 10 points) provided by the respondents to the criteria are calculated and standardized using standard deviation. Each criterion has one frequency value and one impact value, which helps determine its position in the FIA model. NRMs are developed using the DEMETAL method to present the causal relationship and the degree of impact between the dimensions and barriers in a complex system, which can facilitate the decision-making process. Criteria with high ( d  +  r ) values have strong relationships with other criteria, whereas those with low ( d  +  r ) values have weak relationships with other criteria. Furthermore, criteria with a positive ( d  −  r ) can influence other criteria, whereas those with a negative ( d  −  r ) have a high chance of being influenced by others.

Based on the reliability and validity analysis, the Cronbach's alpha of the primary dimension is 0.959, the Cronbach's alpha of UA is 0.886, the Cronbach's alpha of TC is 0.872, the Cronbach's alpha of UE is 0.906, and the Cronbach's alpha of OC is 0.871. The results show that the research questionnaire demonstrates high reliability ( Table 4 ).

Reliability and validity analysis.

Note. Cronbach's α values show that α  < 0.35 is lowly creditable, 0.35 <  α  < 0.7 is moderately creditable, and α  > 0.7 is highly creditable.

4.2.1. Primary Dimensions

Primary dimensions, including UA, TC, UE, and OC, were analyzed in the FIA model, which is characterized by levels of impact and frequency. Figure 3 and Table 5 reveal that TC has a high impact and high frequency; thus, priority should be given immediately. OC and UA have a high impact but low frequency; thus, considerable resources and efforts should be invested to abolish factors. Finally, UE has a low impact and low frequency; thus, developers can maintain standstill action. Therefore, developers should prioritize the following order of the dimensions: TC ⟶ OC ⟶ UA ⟶ UE. As for the NRM model, Figure 4 and Table 5 indicate that UA demonstrates the highest ( d  +  r ) value and the strongest connection with the other dimensions. Furthermore, UA has a positive ( d  −  r ) value, and thus has a remarkable impact on the other dimensions. For further observations on the causal relationships between the primary dimensions, Table 6 provides data on their net influence. Table 6 points out that OC influences all the other dimensions, TC influences UA and UE, and UE influences UA.

Main dimensions' FIA model of medical device development.

Main dimensions' NRM model of medical device development.

Statistical analysis and strategy for main dimensions.

Note. L stands for “low” and H stands for “high.”

Net influence matrix for primary dimensions.

Figure 4 shows that four improvement pathways exist via NRM analysis, that is, OC ⟶ UA, OC ⟶ TC ⟶ UA, OC ⟶ UE ⟶ UA, and OC ⟶ TC ⟶ UE ⟶ UA. Tables ​ Tables5 5 and ​ and6 6 demonstrate that the ranking of the FI is TC > UE > OC > UA, and the ranking of the II is TC > UA = OC > UE. To find a possible pathway, a dimension with a high rank is used to affect a dimension with a low rank. For example, in FI, the second pathway, that is, TC (ranked 1) can improve UA (ranked 4), and this pathway will be accepted. The remaining pathways also follow this logic. Four solvable pathways exist in the FI, and four solvable paths likewise exist in the II. Next, we find four overlapping solvable pathways, as shown in Table 7 . Table 7 summarizes the improvement paths and recommended pathways that medical device developers can follow to solve the main dimensions of medical device development.

• Developers should efficiently take investing resources to improve OC to determine UA.
• Developers should efficiently take investing resources to improve OC, then take priority action to ameliorate TC to determine UA.
• Developers should efficiently take investing resources to improve OC, then take standstill action to define UE to determine UA.
• Developers should efficiently take investing resources to improve OC, take priority action to ameliorate TC, then take standstill action to define UE to determine UA.

Recommended pathways for solving main dimensions.

4.2.2. User Action Dimensions

Four categories comprised of UA, namely, the user needs considerations (UA1), training course (UA2), empirical cognitive ability (UA3), and physical and mental health (UA4). Table 8 and the FIA model in Figure 5 indicate that UA1 and UA2 have a high frequency and high impact. Hence, developers should prioritize solving these two categories immediately. As UA3 has a high impact but low frequency, developers can assess the investment of resources. Finally, the low levels of impact and frequency of UA4 suggest standstill action. Developers are recommended to prioritize the following order of the factors: UA1 ⟶ UA2 ⟶ UA3 ⟶ UA4.

FIA model for UA.

Statistical analysis and strategy for UA.

As for the NRM model, Figure 6 reveals that UA4 influences all the other UA categories, UA1 influences UA3 and UA2, and UA3 influences UA2.

NRM model for UA.

Figure 6 and Table 8 show that the ranking of the FI is UA1 > UA2 > UA3 > UA4, and the ranking of the II is UA1 > UA3 > UA2 > UA4. Two solvable pathways are observed in the FI, and three solvable paths are seen in the II. Next, we find two overlapping recommended pathways: UA4 ⟶ UA1 ⟶ UA2 and UA4 ⟶ UA1 ⟶ UA3 ⟶ UA2.

4.2.3. Technical Capability Dimensions

There were four categories that comprised TC, namely, user interface design (TC1), competitive products (TC2), calibratable maintenance (TC3), and label warning (TC4). Table 9 and the FIA model in Figure 7 indicate that TC1 and TC4 have a high frequency and high impact. Hence, developers should prioritize solving these two categories immediately. As TC2 has a high impact but low frequency, developers can assess the investment of resources. Finally, the low impact and high frequency of TC3 suggest the suspension of action. Developers are recommended to prioritize the following order of the factors: TC1 ⟶ TC4 ⟶ TC2 ⟶ TC3. As for the NRM model, Figure 8 reveals that TC1 influences all the other TC categories, TC2 influences TC3 and TC4, and TC3 influences TC4.

FIA model for TC.

NRM model for TC.

Statistical analysis and strategy for TC.

Figure 8 and Table 9 demonstrate that the ranking of the FI is TC4 > TC3 > TC1 > TC2 and the ranking of the II is TC1 > TC2 > TC4 > TC3. Two solvable pathways exist in the FI and four solvable paths exist in the II. We find two overlapping recommended pathways: TC1 ⟶ TC2 ⟶ TC4 and TC1 ⟶ TC2 ⟶ TC3 ⟶ TC4.

4.2.4. Use Environment Dimensions

UE is comprised of four categories, namely, intended location (UE1), public safety protection (UE2), hygiene requirements (UE3), and device usage time (UE4). Table 10 and the FIA model in Figure 9 indicate that UE1 and UE4 have a low impact but high frequency, thereby suggesting the suspension of action. The low levels of impact and frequency of UE2 and UE4 suggest that standstill action should be taken. Developers are recommended to prioritize the following order of the factors: UE1 ⟶ UE4 ⟶ UE2 ⟶ UE3. As for the NRM model, Figure 10 reveals that UE1 influences all the other usage barriers, UE4 influences UE2 and UE3, and UE2 influences UE3.

FIA model for UE.

NRM model for UE.

Statistical analysis and strategy for UE.

Figure 10 and Table 10 demonstrate that the ranking of the FI is UE4 > UE1 > UE2 > UE3, and the ranking of the II is UE3 > UE1 > UE2 > UE4. Three solvable pathways are seen in the FI, and three solvable paths exist in the II. We found three overlapping recommended pathways: UE1 ⟶ UE4 ⟶ UE3, UE1 ⟶ UE2 ⟶ UE3, and UE1 ⟶ UE4 ⟶ UE2 ⟶ UE3.

4.2.5. Organizational Characteristic Dimensions

OC had four categories, namely, management culture (OC1), team communication (OC2), resource allocation (OC3), and regulatory standards (OC4). Table 11 and the FIA model in Figure 11 indicate that OC4 and OC2 have a high frequency and high impact. Hence, developers should prioritize solving these two categories immediately. The low levels of impact and frequency of OC1 and OC3 suggest that standstill action should be taken. Developers are recommended to prioritize the following order of the factors: OC4 ⟶ OC2 ⟶ OC3 ⟶ OC1. As for the NRM model, Figure 12 provides data on their net influence. Table 11 presents that OC1 influences all the other OC1 categories, OC2 influences OC4 and OC3, and OC4 influences OC3.

FIA model for OC.

NRM model for OC.

Statistical analysis and strategy for OC.

Figure 12 and Table 11 demonstrates that the ranking of the FI is OC4 > OC2 > OC3 > OC1, and the ranking of the II is OC4 > OC2 > OC3 > OC1. Three solvable pathways exist in FI, and three solvable paths are observed in II. We identified three overlapping recommended pathways: OC1 ⟶ OC2 ⟶ OC3, OC1 ⟶ OC4 ⟶ OC3, and OC1 ⟶ OC2 ⟶ OC4 ⟶ OC3.

Recommended pathways based on the results of each of the above factors and tables and planning for the overall improvement path are shown in Table 12 . The recommended improvement pathway order is OC ⟶ TC ⟶ UE ⟶ UA. Medical device developers should examine and analyze each factor. A range of informal and formal organizational processes can influence user considerations, user interfaces, and UE in the development of medical devices. Moreover, adopting a formal decision-making process can help medical device developers develop an integrated and reflective approach to improve business decisions and quality end products.

Summary of improvement pathways.

5. Discussion

This study clarifies the key factors of the medical device development process from the user and stakeholder perspectives. In addition, this study analyzes the frequency and impact of these critical factors on the medical device development process and suggests strategies for improvement.

5.1. Theoretical Implications

This study fills the research gap in key factors that affect the development of medical devices and improvement paths. The FIA results indicate that medical device development stakeholders consider OC as the focus of medical equipment development. Consistent with the views of medical device regulatory authorities, the results demonstrate the importance of the safety and effectiveness of medical devices, including well-designed human-computer interface interaction based on user needs and conditions, and clearly defined product use information that includes mentioning warnings (based on product functions). Medical devices must comply with regulations before they can be marketed, and a recall mechanism should be in place in case of efficacy and safety concerns after marketing. Inappropriate medical devices will be deregulated or banned from the market because they often cause harm to end-users [ 41 ].

Although medical device development stakeholders consider UA and OC as infrequent problems, the two dimensions nonetheless exerts a large impact on medical devices. Numerous studies have noted that shortcomings still exist in the design of medical devices in terms of usability from users' perspectives, such as balancing conflicting user needs and ethical privacy [ 4 , 48 ]. Medical device developers should identify priority input considerations as early as possible to satisfy user needs and provide education, training, and safety guidelines from the users' perspective [ 49 ]. This recommendation is also consistent with stakeholders' views that UA considerations should focus on satisfying user needs and improving training courses, whereas the assessment of user background and individual physical and mental status is difficult and not a priority in the product development process. OC aspects, including regulatory standards and team communication, from medical device design to market entry, are also important in the development and profit of medical devices. Team communication within organization is critical [ 50 ]. Moreover, owing to the particularity of medical devices, regulatory standards have become a key consideration in the marketing of medical devices. Medical device enterprises should familiarize themselves with national regulations as well as the economic status and social backgrounds of their targeted market as early as possible [ 9 ] and develop their product listing process and market plans. Finally, medical device company stakeholders believe that UE is not a priority in the product development process. This finding may be due to the strict regulatory mechanisms of medical devices for product safety specifications; thus, parameter settings and range have been applied to most environmental factors.

5.2. Practical Implications

Medical devices make resource input in its development process far higher than that of general products. Compared with large enterprises, SMEs are disadvantaged in terms of risk control, manufacturing, and operation performance owing to insufficient resources [ 51 ]. This finding has made it necessary for numerous SMEs that manufacture medical devices to evaluate resource planning strategies and develop appropriate paths for product development and healthcare benefits in the context of limited resources [ 52 , 53 ]. Based on the views of stakeholders on the development of medical devices, this study proposes development order and path suggestions in the development of medical devices ( Figure 13 ). The results of this study suggest that medical device development strategies should improve management culture and team communication within the organization and allocate development resources after confirming medical device regulatory standards. After confirming the feasibility of development, medical device developers need to consider the user interface design in terms of technical capability, establish calibration and maintenance standards, and label warnings on their products. Next, medical device developers need to consider the impact of the surrounding environment on the device, including the location, time of use, protective facilities, and hygiene requirements. Finally, even though medical regulations have established the safety and usability of medical devices, medical device developers must still take into consideration the unique circumstances of possible users.

Medical device development strategy.

The development of medical devices is usually for start-up teams or SMEs. The establishment of a climate of intense collaboration and communication between different areas of the organization not only facilitates motivated new projects and rapid decision-making, but also focuses on user needs from concept to disposal of the product lifecycle and integrates the development process, which is an important basis for the development of medical devices [ 54 ]. Due to the complexity of the multisite, multiperson, and multidevice context of many medical interactions, the ensuing user behaviour can have a range of implications for the effectiveness of medical procedures. Developers need to design user interfaces based on technical features, in particular to understand the ergonomic impact of products and clinician/nursing staff interactions with patients based on information from competing products, and to establish product maintenance and warning standards [ 55 ]. The clinical environment usually involves at least two participants in the interaction (clinician and patient) and there are often many complex environmental factors that affect the overall procedure or task, such as the conditions of use of the device (e.g., portability, manoeuvrability, conflict of existing equipment and use of power outlets), the physical environment (e.g., the impact of bedrail design on patient behaviour), and the size of the space available may all limit the usability of the medical device. Developers should therefore also consider the impact of environmental factors on the use of the device when assessing the overall outcome of the device design. Finally, although all of the above factors are met, a medical device is considered marketable. However, the findings of this study suggest that it would be helpful if the development team could take into account the user's condition, including physical and mental health, needs, and training. For example, the packaging of disposable devices may affect the time and efficiency pressures on medical staff, while the sensitive clinical nature of ultrasound is crucial to the physical and psychological comfort of patients. In addition, in the case of long-term health outcomes, other factors of the patient (age, clinical condition, and medication effects) may be no less influential than the design of the device [ 55 ]. This inside-out process of influence is a key in the development of numerous enterprises [ 5 , 56 ] and covers various fields. Thus, in the present study, the proposed development paths of medical devices from the perspective of stakeholders can be seen as logical and valuable.

6. Conclusion

This study uses content analysis and FIA-NRM to discuss stakeholders' views on key factors in medical device development. This study summarizes important factors in the development of medical devices and the views of various stakeholders. This research suggests that the development of medical equipment should start with OC and strengthen TC. Next, according to the evaluation indicators of this study, medical device developers can consider UA and UE strategies and improve functional design, product safety, and clinical application planning with optimal resource allocation. In the future, this study will be able to incorporate input from other stakeholders, including healthcare providers, venture capitalists, and government agencies. It will also be able to conduct case studies on different medical device categories. In the long run, the medical device development strategies developed in this study can benefit the medical industry, health care policy, and national development.

Acknowledgments

The authors would like to thank the laboratory members Mr. Wei-Zhi Lu, Miss. Hui-Chi Wei, and Hsin-Hui Chiu for their help in coding and questionnaire collection. The authors would like to thank Dr. Shun-Min Wang for his expertise in contributing to the construction of the evaluation framework and the research respondents recruited, which helped to relate our research to real-world practice. This research was supported by the Ministry of Technology and Science under grant numbers 108-2221-E-006-063 and 109-2410-H-006-045-MY2 and the Medical Device Innovation Center (MDIC).

Data Availability

Conflicts of interest.

The authors declare that they have no conflicts of interest.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Barriers in reporting adverse effects of medical devices: a literature review

Affiliations.

• 1 Dasmesh College of Pharmacy, Faridkot, Punjab, India.
• 2 Shri Guru Ram Das College of Medical Sciences and Research, Amritsar, Punjab, India.
• 3 Pharma Innovation Lab, Department of Pharmaceutical Sciences and Technology, Maharaja Ranjit Singh Punjab Technical University, Bathinda, Punjab, India. [email protected].
• PMID: 39259332
• DOI: 10.1007/s00210-024-03431-x

Medical devices play an essential role in the delivery of healthcare but its use is not entirely risk free. There are several instances where it causes mortality or morbidity among users. It is important to evaluate the risks involved at every stage of its application to bring improvement in the standard of healthcare. For the purpose Materiovigilance Program of India was launched on July 6, 2015. Despite these efforts, available data suggests that reporting of adverse events is very low. The present study aims to identify barriers that influence the reporting of adverse events of medical devices and outline a strategy to overcome these barriers. Systemic review method has been adopted to achieve these ends. Thirty-one papers have been selected based on the inclusion criteria related to objective of the study. Lack of awareness, attitude, and resources are found to be major barriers at the individual level for not reporting adverse effects of medical devices. The organizational factors such as hierarchical set up, lack of time and incentives, and furthermore lack of industry responsiveness have been identified as prominent barriers to the reporting of adverse events. In order to improve the reporting level, it is important to make access and contact easier with the reporting system. Engaging healthcare professionals at various levels by acknowledging and appreciating their contribution. The adverse events of medical devices should not be restricted to physicians; only rather other health care professional such as nurses, pharmacists, and technicians should also be encouraged to report any adverse event of medical devices.

Keywords: Adverse events; Barriers; Materiovigilance; Medical devices; Pharmacovigilance; Regulatory affairs.

© 2024. The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.

PubMed Disclaimer

• Akram MF, Manak S, Inder D, Khan MA, Salman MT (2022) Pharmacology teaching in dental education in India: Time for a reappraisal. J Pop Ther Clin Pharmacol 29(3):1–10. https://doi.org/10.47750/jptcp.2022.843 - DOI
• Alshakka M, Badullah W, Al-Dhuraibi A, Alshagga S, Ibrahim MIM (2022) Teaching pharmacovigilance to undergraduate students: Our experience in poor-resource setting. J Pharm Bioal Sci 14(1):31–37. https://doi.org/10.4103/jpbs.JPBS_532_20 - DOI
• Alshamsi H, Almutairi A, Al Mashrafi S, Kalbani T (2020) Implications of language barriers for healthcare: A systematic review. Oman Med J 35:e122–e122. https://doi.org/10.5001/omj.2020.40 - DOI
• Amoore J, Ingram P (2003) Learning from adverse incidents involving medical devices. Nurs Stand Royal College Nurs Great Britain 17(29):41–46. https://doi.org/10.7748/ns2003.04.17.29.41.c3368 - DOI
• Badnjević A, Pokvić LG, Deumić A, Bećirović LS (2022) Post-market surveillance of medical devices: A review. Technol Health Care 30(6):1315–1329. https://doi.org/10.3233/THC-220284 - DOI - PubMed

Publication types

• Search in MeSH
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

• MDR CER Writing
• Systematic Literature Review
• Literature Review Templates
• Full Service PMS
• Vigilance Monitoring and Reporting
• PMCF Surveys Solution
• Clinical Evaluation Doc Reviews
• IVDR Writing
• FDA Submissions
• Consulting Services
• Lit Review Software
• EU MDR Templates for CER and PMS
• Free Downloads
• Featured In …
• Testimonials

Literature Reviews for EU MDR

Want to know what the most commonly criticized document in an MDR submission is?

Your clinical literature review.

Reviewing the literature is essential to the clinical evaluation process for medical devices.

The clinical evaluation report assesses a medical device’s benefit-risk profile to ensure its continued safety, efficiency, and compliance. A critical part of the clinical evaluation report is the literature search and review, which determines the current state-of-the-art, and the behavior of a medical device in the market, while summarizing the clinical data.

The  clinical evaluation report  (CER) demonstrates compliance with the  European Medical Device Regulation  (MDR; 2017/745) for medical devices and  MEDDEV 2.7 rev. 4 .

MEDDEV 2.7 rev 4  describes a medical device’s clinical evaluation process for medical devices and the requirements for the literature search, i.e., searching for relevant literature, compiling research, and explaining how the gathered clinical evidence pertains to the safety and efficiency of your medical device.

Note: This article uses the terms literature search and literature review interchangeably.

Why you need an excellent literature review

For many medical device manufacturers, the literature search represents the majority, if not all, of the clinical data they collect on their devices during device development and subsequent clinical evaluation updates. With it, it is possible to demonstrate and critically evaluate the safety and efficacy of their device during clinical evaluation.

Even for companies that have performed clinical trials, a literature search still ensures a summary of the valuable clinical data regarding the different aspects of their device and its performance.

Lastly, and maybe most importantly, the literature search is mandatory for clinical evaluation reports under the EU MDR and will help determine if your medical device receives a  CE mark .

What is a great literature search?

A good literature search provides the following:

• The context for a problem.
• Its proposed solution.
• A disease/condition or medical process problem.
• The intervention is designed to treat or solve it.

It should quickly summarize the maximum amount of published literature.

A well-organized and thorough literature search provides manufacturers with critical information on who else has attempted to solve this problem, which approaches were taken (similar devices on the market), how they were received in the marketplace, and if they were effective and safe.

It can also reveal to the manufacturer whether other available products may be better suited to solve the problem than the device they’re attempting to put on the market.

Understanding how other devices have been received can help ensure that the new solution will be well received.

For example, suppose a device is designed to address a specific concern shared by many physicians but a few patients. In that case, it might not be as attractive as a similar device that addresses concerns held by both patient groups and doctors.

A thorough literature search will also help you create an excellent systematic process that will set you up for future documentation updates, reducing future workload.

How literature searches can improve other medical device processes

Medical device development.

A literature review serves many purposes in the medical device development process, from informing the R&D team about what has been done before to understanding how a new device fits into the market and helping determine whether a project is worth pursuing.

An example of a good literature review is one that also helps manufacturers communicate their ideas to partners, investors, and other stakeholders.

Reading a good literature review can better understand a device’s value and benefits/risks more than other methods, like reading articles describing the device or watching a video.

Post-market surveillance and post-market clinical follow-up

Although the literature review for clinical evaluations is more about pre-market surveillance, the clinical evidence gathered can significantly help you design your medical device’s post-market surveillance program.

A literature review can help convince regulatory bodies that a new device’s benefit-risk ratio is beneficial enough to approve it.

It also helps support claims made by manufacturers about their products, including their intended use, safety, and effectiveness. Furthermore, it can be used as clinical data in marketing materials, such as brochures and videos.

This is particularly important when introducing an innovative product to the market because it can help healthcare professionals understand how it differs from existing devices.

A document showing how the theoretical device will work in the market or exactly which deficiencies current medical device designs have – which the new device is covering – is beneficial.

Five practical tips for medical device clinical literature searches

Even when everything else goes right, you might not get the relevant results you expected from your literature review.

This can be due to incorrect framing of the research question, needing to understand exactly how your search terms interact with the database, or even problems with choosing suitable databases to search.

So, how can you get the right results in your literature reviews that will support your clinical evaluation and demonstrate the acceptability of your benefit-risk profile?

Choose your databases wisely.

Not all databases are equal, and they will not necessarily yield the same results for your literature review. Therefore, more is needed to search just one – to find articles and research examples relevant to your device, you need to search multiple databases.

For example, using the (unfiltered) search term  “Silk fibroin AND (dermal OR filler OR “dermal filler” OR “soft tissue augmentation”)  will yield three results in PubMed, 16,700 results in GoogleScholar, and 181 results in LIVIVO.

Some of these differences are due to how the search engines work, while others are due to different publications available in each database.

It is essential to understand how each database works and how to phrase your literature review terms to get valuable results in each database.

If you are going after a CE Mark under EU MDR, then you’ll want to select databases that give you a comprehensive and global (with a EU focus) result set.

Be organized.

Some literature reviews will give you a plethora of information, primarily if you work with devices in larger fields, such as cardiovascular and orthopedic surgery. However, it is essential to evaluate all that information critically.

You must set up a straightforward process for conducting your literature search before you start, i.e., you have to write a literature search protocol. You need to have each step in place before you do it – otherwise, it is easy to get lost when you’re halfway through or, even worse, when you’re almost done.

You should always do a preliminary search to get an overview of the information and publications and test the waters of your search terms. Get a feel for the task.

Once you have determined what you are looking for and how to get there, write a detailed plan for the search. Include search terms, inclusion/exclusion criteria, key concepts, the databases you will search and what tools you will use for your search.

You are writing a recipe for your search you can follow when you start. This recipe becomes the literature search protocol.

Use digital tools to make your life easier.

Expanding on the previous step, generating bibliographies and search result files, and storing references, let alone organizing them, is a pain in the butt.

There is a lot of meticulous detail work and moving parts to the literature search, and there is no need to complicate matters for yourself – use as many digital tools to make your life easier as possible. Download your search results, use a citation manager, and ensure you’re using the right programs when reviewing your data results.

The team at CiteMed has spent years refining and perfecting our Systematic Literature Review Software platform so that it’s easy to use, and can save you massive amounts of hours per review project. If you’d like to give it a test drive, contact us today.

Use (Boolean) operators.

Boolean operators are always highlighted when people advise on literature reviews, and for a good reason. The right boolean operator can make or break your literature search (and sanity).

Operators allow you to narrow your literature review to focus on only the publications relevant to you and save you hours. Likewise, your search can be widened if you are not getting enough results.

The most well-known operators are AND, OR, and NOT. Use them wisely.

Use search filters.

Most literature reviews have some search restrictions in place. It can be a time limit, i.e., only doing literature searches for the last five years. It can be a language (no use getting results in Japanese if you can’t understand them), a medical discipline, or the type of publication you are looking for.

Every database will have filters you can use to refine and narrow down your search – make sure to use them as much as possible. It is your most straightforward tool to obtain the results you need.

A complete Protocol and search strategy are defined by our team when working on Literature Search projects. You can read more about them here .

How automation can improve your literature searches and device evaluation

More than 3 million scientific articles are published in English annually, and publication rates are growing by almost 10% annually.

Information overload

This information explosion makes it increasingly difficult for researchers to keep up with the latest field developments. In addition to the apparent challenge of keeping up with new content, there is another problem that many researchers face: understanding the context, methods, and goal of the research being presented. Those can often only be understood by reading papers in detail. Hence, the regulatory team members must read, organize, and report on many articles.

The increasing addition of papers to the scientific community has created information overload. For many regulatory people and medical writers, it is simply impossible to read through all the search results carefully to find relevant information for clinical evaluation reports (CER).

Artificial intelligence and clinical data processing

Artificial Intelligence (AI) is not novel, and AI allows for an excellent collaboration between humans and machines, the very best of which is yet to come. Even so, currently available software can significantly reduce the workload for medical device manufacturers preparing literature reviews and clinical evaluation reports.

Manufacturers have more control over the data screening process and data extraction in an automated process than in a manual process.

Several automated programs with integrated artificial intelligence and machine learning allow users to create labels, filters for the data extraction process, inclusion criteria & research questions customization, and indexing to balance precision and recall.

Some come with summarisation and reporting tools highlighting hidden connections in the textual data. In addition, automated bias assessment and study identification ensure the automated approach only needs a little human intervention beyond the search strategy.

Interactive machine learning systems are more straightforward than one would assume. For example, some software offers data visualization, meta-analysis, evidence synthesis, and text mining, which can all be highly useful during literature screening and clinical data appraisal.

Despite advanced with Artificial Intelligence, we advise our clients to avoid it when performing SLRs for an EU MDR submission. The reason? Because they invite further scrutiny form your Notified Body auditor, and the ever-challenging question of “how can you validate your results?”.

Be Certain that Performing a Literature Review Is Worth Your Time

Any clinician with enough time, tools, and support can draft a protocol and perform an acceptable Literature Review. However, for most Regulatory Affairs and Quality professionals, the question they should be asking is:

Is it really worth the effort?

The reality is that high quality systematic literature reviews take a ton of time to perform, organize, and format. Without the use of top tier tools, they take even longer. Can you really afford to split your focus and spend weeks sorting through duplicate articles and wrangling spreadsheets when the job could be done for you affordably?

Before you dive in and start tackling your own Literature Reviews, we strongly recommend you speak with a member of our team to determine if performing them yourself is really worth your time.

For more reading on our legendary Literature Reviews or to get a quote click here .

Want more EU MDR and Regulatory Insights?

We send weekly emails with the latest regulatory developments, templates, and strategies straight to QA/RA Professionals like you. Sign up below to get access today.

" * " indicates required fields

Step 1 of 4

Want Access to Everything EU MDR?

Join over 3,000 Regulatory and Quality Directors, Engineers and Consultants who receive weekly industry Whitepapers, Templates, and EU MDR news right to their inbox

By signing up, you agree to receive email marketing

• Systematic Review
• Open access
• Published: 10 September 2024

Invitation strategy of vaginal HPV self-sampling to improve participation in cervical cancer screening: a systematic review and meta-analysis of randomized trials

• Ho Yan Wong 1 &
• Eliza Lai-yi Wong 2 , 3 , 4  

BMC Public Health volume  24 , Article number:  2461 ( 2024 ) Cite this article

Metrics details

Human papillomavirus (HPV) self-sampling is recognized as a feasible option for enhancing screening for cervical cancer, particularly among hard-to-reach women. The magnitude of the effectiveness of screening participation under different invitation strategies was reported. This review seeks to compare the effectiveness of invitation strategies in increasing screening participation of HPV self-sampling across diverse study settings.

A systematic literature search was conducted in Embase, MEDLINE, and PubMed in April 2023. Articles were included if (1) their target participants were aged between 25 and 70 years; (2) participants in the intervention arm were randomized to receive HPV self-sampling devices through various invitation strategies; (3) participants in the control arm who either received invitations for cervical cancer screening other than HPV self-sampling or opportunistic screening as usual care; (4) studies that provided sufficient data on screening participation in HPV self-sampling as outcome measured. The study design of the included articles was limited to randomized controlled trials.

A total of 15 articles were included in this review. Invitation strategies of disseminating HPV self-sampling devices included opt-out and opt-in. Meta-analysis revealed screening participation in the self-sampling group was significantly greater than control arm (OR 3.43, 95% CI 1.59–7.38), irrespective of the invitation strategy employed. Among invitation strategies, opt-out appeared to be more effective on increasing screening participation, compared to control and opt-in strategy (opt-out vs. control OR 3.91, 95% CI 1.82–8.42; opt-in vs. control OR 1.34, 95% CI 0.28–6.39).

Conclusions

Opt-out strategy is more successful at improving screening participation compared to opt-in and routine invitation to cervical screening. It is therefore a promising way to improve participation in cervical cancer screening. The findings of this review provide important inputs to optimize strategies for inviting women to participate in vaginal HPV self-sampling across the study setting, thus improving participation in cervical cancer screening.

Peer Review reports

Cervical cancer is a highly preventable malignancy among women with vaccination and regular screening. The nationwide cervical screening programs in developed countries has led to a significant decline in both of its incidence and mortality. [ 1 , 2 ] However, it is still ranked the fourth most common female cancer in the world due to low screening coverage. [ 2 , 3 ] Suboptimal screening participation remains a global challenge to the success of screening programme and the majority of cervical cancer occur in women who are either never-screened or underscreened [ 3 ]. The causal relationship between persistent infection with high-risk human papillomavirus (hrHPV) and the development of cervical cancer was well established [ 4 ], Human papillomavirus (HPV) testing has therefore become an emerging option for cervical cancer prevention in the past decade as the timely detection of hrHPV infection enables early management of possible cervical intraepithelial lesions and prevent progression of cervical cancer [ 1 , 5 ]. HPV testing is performed through clinician-collected or self-collected; the latter is known as vaginal HPV self-sampling which involves the use of a self-sampling device for collecting cervicovaginal samples by women themselves.

Recently, vaginal HPV self-sampling is promoted to tackle the barriers to conventional cytology and clinician-collected sampling, such as psychological distress, invasion of privacy, cost of attending clinic [ 3 ]. Studies have shown that the sensitivity of vaginal HPV self-sampling is similar to clinician-collected sampling in detecting cervical intraepithelial neoplasia grade two or worse (CIN2+) when polymerase chain reaction (PCR) based tests are used [ 5 , 6 , 7 ]. The demonstrated accuracy of vaginal HPV self-sampling underpins its potential by adopting HPV testing as primary screening test [ 3 ]. Compared to clinician-collected HPV testing, women offered vaginal HPV self-sampling have shown greater participation, particularly among hard-to-reach populations [ 8 , 9 , 10 ]. A systematic review in 2024 showed there was a high acceptability, feasibility and sustainability by adopting vaginal HPV self-sampling as an alternative cervical cancer screening tool [ 11 ]. Besides, local studies have suggested that the low awareness and literacy of HPV self-sampling was the possible hurdle to increase the screening participation [ 12 ]. Invitation strategies of vaginal HPV self-sampling are therefore essential for successful cervical screening programme. Existing invitation strategies were generally categorized into “opt-out” and “opt-in”. The former was also known as “send-to-all”, involving direct dissemination of self-sampling devices to all target women for cervicovaginal sampling, whilst the latter required women to indicate their request to obtain sampling devices. Current evidence revealed that various invitation strategies for disseminating HPV self-sampling devices resulted in substantial variation in screening participation. Among studies involving multiple arms for various invitation strategies, it revealed that opt-out strategy potentially resulted in higher screening participation of cervical screening, compared to opt-in [ 13 , 14 ]. This study aimed to assess the impact of invitation strategies on screening participation and to identify the most effective strategy on increasing screening participation across the study settings. The findings provide insights into informing the tailored policy of cervical cancer prevention and combating cervical cancer through improving its screening participation.

Search strategy and selection criteria

A systematic literature search was conducted through electronic databases, including Embase, MEDLINE and PubMed, in April 2023. Specific search terms and MeSH headings were employed in individual databases to ensure a robust search of the literature and capture the conceptual ideas of the research topic. The search terms included HPV, cervical cancer, screening, DNA testing, self-sampling, self-collection specimen, uptake, participation and coverage. The full search strategy was shown in appendix A . Studies were considered when the following eligible criteria were fulfilled: women aged between 25 and 70 years; participants in the intervention arm who were randomized to receive self-sampling devices through various invitation strategies for vaginal HPV testing; participants in the control arm who either received invitations for cervical cancer screening other than vaginal HPV self-sampling or opportunistic screening as usual care; studies that provided sufficient data on screening participation in vaginal HPV self-sampling as outcome measured. The study design of the included studies was restricted to randomized controlled trials as it is the most robust for establishing causal relationships. The inclusion criteria were not limited by the study setting or language. To ensure the inclusion of the most recent evidence on the effectiveness of HPV self-sampling for improving screening participation, only articles published from 1 January 2013 to 31 December 2023 were considered. Two authors independently reviewed titles and abstracts to identify studies for full-text screening. The full text of all studies independently reviewed for their eligibility. Throughout the screening process, discrepancies in the screening decisions between the two authors were resolved by discussion and consensus. Pre-specified analysis was adopted in this review by outlining the objectives, outcomes and anticipated statistical procedures in this review prior to the implementation. Random-effects meta-analysis was used to estimate the effect of screening participation across different invitation strategies. All statistical analyses were performed using Review Manager (RevMan). Cochrane Risk of Bias 2 (RoB 2) tool for randomized trials was adopted for quality assessment of the included studies [ 15 ].

Research question

In this review, screening participation of HPV self-sampling refers to the return of collected cervicovaginal specimens within the designated period of an individual study, while invitation strategy is defined as the process of disseminating materials to facilitate women’s participation in HPV self-sampling. Our research question is as follows: “Which of the invitation strategies are considered the most effective on increasing participation in cervical cancer screening with respect to the study setting?” This systematic review was presented in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement 2020 [ 16 ].

We identified a total of 477 records, including 466 records from electronic databases and 11 records from manual searching. There were 154 relevant records extracted after titles and abstracts screening. Seventy-eight records were excluded for irrelevant variables according to the research question, including not provide sufficient data on screening participation of HPV self-sampling ( n  = 27), study designs other than randomized controlled trials ( n  = 48), urine HPV self-sampling ( n  = 3). At the completion of full-text screening, sixty-one records were further excluded as 48 of those did not fulfil eligibility criteria of this review and the remaining 13 articles did not report specific data on screening participation of HPV self-sampling. A total of 15 articles were eventually included in this review [ 13 , 14 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ]. The flowchart of the retrieval of records in the PRISMA diagram was presented (Fig.  1 ).

Study characteristics

The 15 studies included 94,908 participants, with individual sample sizes ranging from 400 to 19,851. All the included studies were published between 2013 and 2022. Twelve of these studies were conducted in high-income countries, including six each in European and Western countries. The latter included Argentina, Australia, Canada, United Kingdom and United States. The remaining three studies were conducted in low- and middle-income countries (LMICs) which were Ethiopia, Nigeria and Uganda. Studies adopting opt-out strategy were primarily conducted in Western countries ( n  = 5, 33.3%), following by Europe ( n  = 3, 20.0%) and LMICs ( n  = 3, 20.0%). Conversely, there was only one study in Western countries adopted an opt-in approach only [ 20 ]. Studies with multiple arms involving both opt-out and opt-in approaches were exclusively conducted in Europe ( n  = 3, 20.0%). Of the included studies, participants from 10 studies (66.7%) targeted nonattenders, under screened and never-screened, while participants in the remaining 5 studies (33.3%) were recruited from general population. One study conducted in LMICs targeted women with low socioeconomic status [ 21 ]. The age of the study participants ranged from 25 to 70 years. The mean age of participants from 6 of the included studies was 47.16 while the remaining studies reported proportion of participants in each age group.

Study flow chart. The process of article selection was illustrated in accordance with the PRISMA flow diagram 2020. A literature search was conducted in selected electronic databases and manual searching, with 477 records initially identified. Fifteen randomized trials were included in the review

All participants in the self-sampling arm were provided with HPV self-sampling devices through various invitation strategies, including opt-out (send-to-all) and opt-in. Over 70% of the included studies ( n  = 11, 73.3%) employed opt-out strategy by directly offering participants with self-collection devices through direct mail ( n  = 6, 54.5%) or door-to-door method ( n  = 4, 36.4%) by research assistants, community health workers, healthcare professionals, or outreach workers. However, one study unclearly reported how the participants in self-sampling arm were provided with devices under opt-out strategy [ 21 ]. Only one study (6.67%) employed opt-in strategy, where participants are required to fill in postal order form to obtain the devices [ 20 ]. Additionally, three studies (20%) involved multiple arms, by including opt-in and opt-out strategies into intervention arms separately. Of these opt-out arms, devices were exclusively offered to participants through direct mail while participants in control arm were provided with various channels for placing their orders, including online platforms (email and website), mail, phone and short message service. Among the included studies, the opt-out strategy was identified as the most common method for disseminating self-sampling devices. The characteristics of included studies were presented in Table  1 .

Various types of self-sampling devices were utilized, including swabs [ 17 , 20 , 21 , 22 , 23 , 24 , 25 , 27 , 28 ] ( n  = 9, 60%), brushes [ 13 , 17 , 18 , 19 , 26 ] ( n  = 5, 33.3%), and lavage-based devices [ 18 ] ( n  = 1, 6.67%). Among these, one study employed more than one type of sampling device [ 18 ]. Nevertheless, one study did not explicitly specify the type of sampling device used [ 29 ]. Of the included studies, women from self-sampling arms primarily performed self-sampling at home, except for one study where women performed in a private area of a health post [ 19 ]. Participants in the control arm were primarily provided with invitation for cytology screening as usual care. One study conducted in LMICs provided participants with the option for hospital-based sampling by trained nurses [ 21 ] whereas women from two other studies conducted in LMICs were offered with appointments for visual inspection with acetic acid (VIA) [ 19 , 22 ]. The length of measuring screening participation of HPV self-sampling ranged from 3 to 12 months. Five of the included studies (33.3%) assessed participation of cervical cancer screening within 6 months after invitation letter was sent [ 13 , 17 , 25 , 26 , 28 ] whilst six studies accepted screening participation within the study period [ 19 , 22 , 23 , 24 , 27 , 29 ]. Only two studies measured screening participation at multiple time points [ 20 , 25 ].

The overall methodological quality of the 15 included studies was moderate to good according to the RoB 2 tool [ 15 ]. The risk of bias in the randomization process was generally low, while four studies (26.7%) were categorized as some concerns. One study mentioned to adopt Zelen’s design for randomization without detailed discussion [ 20 ]. Contamination between treatment groups likely occurs as participants were fully aware of treatment allocation. Another study did not include details of randomization and selection bias would be a concern [ 24 ]. Simple randomization was employed to two intervention arms in 1:1 ratio in one study. However, the allocation concealment and blinding were unclearly discussed [ 23 ]. All included studies were deemed low risk in remaining four domains, including deviations from the intended interventions, missing outcome data, measurement of the outcome and selection of the reported result. The findings of the quality assessment of the included studies and reported outcomes are presented in Table  2 .

Impact of invitation strategy on screening participation

Fourteen of the included studies (93.3%) compared the effectiveness of opt-out strategy on disseminating devices between self-sampling arms and control arms. Of these studies adopting opt-out strategy, irrespective of the approach of disseminating devices, it was shown that participants in self-sampling arm generally demonstrated greater screening participation than control arm. The pooled odds ratio of screening participation between self-sampling arm and control arm under opt-out strategy was 3.91, 95% CI 1.82–8.42 (Fig.  3 ), indicating participants in self-sampling arms receiving devices under opt-out strategy were approximately 4-fold greater in the likelihood of performing cervical cancer screening compared to control arms. Under opt-out strategy, the dissemination approach can be further categorized into direct mailing and door-to-door. The pooled odds ratio of screening participation between self-sampling arm and control arm was 2.28, 95% CI 1.03–5.04 (direct mailing vs. control) and 9.64, 95% CI 2.51-37.00 (door-to-door vs. control) respectively (Figs.  4 and 5 ). Door-to-door dissemination through opt-out strategy was therefore demonstrated to be more effective in increasing screening participation compared to direct-mailing.

Four of the included studies compared the effectiveness of opt-in strategy on disseminating devices between self-sampling arm and control arm. Regardless of the availability of channels for ordering devices, screening participation among women in self-sampling arm were generally greater than control arm. The pooled odds ratio of screening participation between self-sampling arm and control arm under opt-in strategy was 1.34, 95% CI 0.28–6.39 (Fig.  6 ). As the confidence interval crossed 1, there was no statistically significant difference in screening participation between opt-in arms and control arms.

Forest plot for the odds ratios of screening participation between self-sampling arm and control arm. The overall pooled effect of 15 studies was OR3.43, 95% CI 1.59–7.38, indicating that women from self-sampling arm receiving devices through various invitation strategies were 3.4-fold greater in the likelihood of performing cervical cancer screening than control arm

Forest plot for the odds ratios of screening participation between opt-out arm and control arm. The overall pooled effect of 14 studies was OR3.91, 95% CI 1.82–8.42, indicating that women from self-sampling arm receiving devices through opt-out strategy were approximately 4-fold greater in the likelihood of performing cervical cancer screening than control arm

Forest plot of the odds ratios of screening participation between direct-mailing (opt-out) arm and control arm. The overall pooled effect of the 9 studies was OR2.28 (95% CI 1.03–5.04), indicating that women from direct-mailing arm receiving devices through opt-out strategy were approximately 2.3-fold greater in the likelihood of performing cervical cancer screening than control arm

Forest plot of the odds ratios of screening participation between door-to-door (opt-out) arm and control arm. The overall pooled effect of the 4 studies was OR9.64 (95% CI 2.51-37.00), indicating that women from door-to-door arm receiving devices through opt-out strategy were 9.6-fold greater in the likelihood of performing cervical cancer screening than control arm

Forest plot of the odds ratios of screening participation between opt-in arm and control arm. The overall pooled effect of the 4 studies was OR1.34 (95% CI 0.28–6.39), indicating thatthere was no statistically significant difference in screening participation between opt-in arm and control arm

Impact of HPV self-sampling on screening participation

Women in self-sampling arm generally demonstrated greater screening participation than women in the control arm, except for the findings of one study conducted by Zehbe, Jackson [ 27 ] which reported that there was no significant difference in screening participation ( p  = 0.628) between the self-sampling arm and the control arm, with values of 20.0% and 14.3%, respectively. Among the included studies, the pooled odds ratio of screening participation in self-sampling arms compared to control arms was 3.43, 95% CI 1.59–7.38 (Fig.  2 ), indicating that the likelihood of performing cervical cancer screening of women in the self-sampling arms was 3.4-fold greater than the control arms.

Impact of demographics on screening participation

Ten of the included studies (66.7%) reported socioeconomic data, including income level, employment status and health insurance status of participants as baseline demographics. Regarding the predictors of screening participation of cervical screening services, Modibbo, Iregbu [ 18 ] indicated that socioeconomic status and educational level were not significantly associated with screening participation, with reported p  values of 0.861 and 0.894, respectively. Of the included studies, 12 studies recruited participants aged 30 years or above, while participants from the remaining 3 studies were aged 25 years or above. The pattern of screening participation of HPV self-sampling across age groups varied. Nearly half of included studies ( n  = 7, 46.7%) reported middle-aged women in self-sampling arms aged between 40 and 60 years demonstrated the highest participation [ 13 , 18 , 20 , 21 , 23 , 24 , 26 ]. Considering the study setting, over 70% of the included studies ( n  = 11, 73.3%) were conducted in high-income settings, 1 in middle-income setting, 3 in LMICs and low-income setting according to the classification of the World Bank [ 30 ]. The findings of studies conducted in high-income settings revealed that women in self-sampling arms generally demonstrated greater screening participation than control arms, except for four studies [ 18 , 20 , 26 , 28 ]. Studies conducted in LMICs and low-income settings showed screening participation in self-sampling arms was consistently greater than control arms. Vaginal HPV self-sampling was therefore found as a feasible option to increase participation of cervical screening services, especially in resource-constraint settings.

Impact of educational materials on screening participation

More than half of the included studies ( n  = 9, 60%) provided women in self-sampling arms with educational materials, including leaflet, pictorial instructions and video, whereas four studies (26.7%) did not explicitly report the use of educational materials for self-sampling arms. Of these educational materials, pictorial instructions were the most commonly used. One study conducted by Winer, Lin [ 28 ] did not provide description of educational materials used. In addition to educational materials, two studies offered onsite education by community health workers to women in self-sampling arms [ 17 , 22 ]. Both studies reported that women in self-sampling arms showed significantly greater screening participation compared to other studies involving educational materials. Besides, one study in LMICs involved direct supervision by trained healthcare practitioners to women during self-sampling procedure, with a statistically significant difference in screening participation ( p  < 0.001) between self-sampling arm and control arm [ 19 ].

Cervical cancer is a largely preventable cancer in females by vaccination and regular screening, however, it remains a significant public health concern especially in LMICs. Although the incidence and mortality of cervical cancer has been substantially declined due to the introduction of national screening programmes. screening participation is still suboptimal. In response to this challenge, vaginal HPV self-sampling has been suggested as a feasible alternate of cervical cancer screening to improve the screening participation. In accordance with the global strategy to eliminate cervical cancer launched by the World Health Organisation (WHO) [ 31 ], HPV testing has been proven as a superiority than conventional cytology or VIA, in term of its sensitivity and reproducibility. HPV testing has comparable accuracy in detecting cervical intraepithelial neoplasia grade two or worse (CIN2+). Despite the proven accuracy and feasibility of HPV self-sampling in increasing screening participation, worldwide use of HPV self-sampling remains very low. Until 2021, there was solely 48 countries adopting HPV-based screening as primary screening approach and majority of these countries are situated in Europe and America [ 3 ]. In addition, recent systematic reviews clearly demonstrated that HPV self-sampling is an acceptable method of cervical cancer screening in reaching never screened and under-screened women as it overcomes those barriers in conventional cytology, namely psychological distress, discomfort, inconvenience and privacy [ 7 , 32 ].

The current findings suggested that screening participation in HPV self-sampling arms was generally greater than control arms when women were provided with self-sampling devices, regardless of the invitation strategy employed. Considering the effectiveness of invitation strategies on screening participation, opt-out strategy appears to bemore effective on reaching women and increasing their screening participation compared to control and opt-in. This finding aligned with previous findings from several recent systematic reviews by Yeh, Kennedy [ 33 ] and Costa, Verberckmoes [ 34 ]. Among opt-out strategy, direct mailing of self-sampling devices was found as the most common approach in high-income settings. It is likely attributed to the organized postal system that enables effective and reliable delivery to women living in geographically diverse area. On the contrary, all included studies conducting in LMICs, women in self-sampling arm were offered devices through door-to-door approach. It can be explained by their collected specimens rely on the assistance of healthcare workers for transportation in the rural settings. The highest screening participation was observed when women received self-sampling devices from outreach workers providing onsite instruction with standard script and diagram through door-to-door approach [ 22 ]. It can be possibly explained by the face-to-face situation enables women to directly clarify their concerns regarding self-sampling procedures. It also allows outreach workers to provide women with reassurance and support during the interaction which makes women less likely in the failure of completing self-sampling procedures. Although substantial heterogeneity across included studies was observed, vaginal HPV self-sampling had generally positive impact on screening participation irrespective of the invitation strategy. These findings further support existing evidence on the feasibility of vaginal HPV self-sampling as the alternate option of cervical cancer screening to increase its screening participation. Besides, nine of the included studies in high-income settings disseminated self-sampling devices to home addresses of participants through direct mail [ 13 , 14 , 18 , 23 , 24 , 25 , 26 , 27 , 28 ], their findings consistently demonstrated that screening participation in self-sampling arm significantly greater than the control arm. It implies that direct mailing is acceptable and effective approach to reach women in increasing screening participation. Although screening participation involves a complex process of health maintenance behaviour, surprisingly, none of the included studies measured screening participation in the longitudinal manner. Only one study assessed screening participation in HPV self-sampling at 12 months [ 20 ]. Currently, limited longitudinal studies pinpointed the compliance of HPV self-sampling as a regular cervical screening approach. It provides insights on examining the feasibility of HPV self-sampling for sustaining regular screening behaviour in the future studies. This review compares the effectiveness of invitation strategies of disseminating self-sampling devices on improving screening participation across the study setting. It ascertains vaginal HPV self-sampling is a feasible way to improve participation in cervical screening, especially when self-sampling devices were disseminated by door-to-door approach through opt-out strategy.

Several limitations should be addressed in this review. Majority of studies included various types of educational materials, namely written instructions with diagrams on self-sampling procedures, while two of the included studies involved direct supervision of women in the intervention arms during self-sampling procedures. Nevertheless, variation of its impact on screening participation has not been clearly discussed. Further studies appraising the external factors associated with screening participation are therefore necessary. The considerable statistical heterogeneity across the studies implied that the results of individual studies were poorly overlapped which leaded to substantial variation of screening participation reported by the included studies. In addition, little is known about the experiences of end users during self-sampling procedures. This study provides insights for future qualitative studies to explore the experience of vaginal HPV self-sampling across female population. Our strengths in this review were reflected by the inclusion of rigorously conducted studies. There was also no restriction on language or study setting to ensure our extensive search on this topic.

In conclusion, the findings of this review underpin that opt-out appears to be more effective on increasing screening participation, compared to control and opt-in strategy. To enhance the generalizability of the findings, pilot intervention studies should be conducted to address the viability and appropriateness of the invitation strategy of vaginal HPV self-sampling to the target population and study setting. Various invitation strategies for the dissemination of HPV self-sampling devices were adopted and presented with a substantial impact on screening participation. Women receiving devices through opt-out strategy appears to have greater screening participation than women from opt-in arms and usual care. Follow-up studies assessing the feasibility of vaginal HPV self-sampling for maintaining compliance with cervical screening are warranted as current literature has primarily examined screening behaviour on the cross-sectional basis. Finally, future qualitative studies exploring the experience and attitudes towards HPV self-sampling among the end-users are valuable. The findings of this review provide important inputs to optimize invitation strategies of vaginal HPV self-sampling across the study setting and thus improving participation in cervical cancer screening.

Data availability

Data is provided within the manuscript or supplementary information files.

Abbreviations

Atypical squamous cells of undetermined significance

Community health worker

Confidence interval

Cervical intraepithelial neoplasia grade two or worse

Deoxyribonucleic acid

High-risk human papillomavirus

Human papillomavirus

Low- and middle-income countries

Polymerase chain reaction

Preferred Reporting Items for Systematic reviews and Meta-Analyses

Risk of Bias

Visual inspection with acetic acid

World Health Organisation

Arbyn M, Weiderpass E, Bruni L, et al. Estimates of incidence and mortality of cervical cancer in 2018: a worldwide analysis. Lancet Global Health. 2020;8(2):e191–203.

Article   PubMed   Google Scholar  

Cohen PA, Jhingran A, Oaknin A, et al. Cervical cancer. Lancet. 2019;393(10167):169–82.

Serrano B, Ibáñez R, Robles C, Peremiquel-Trillas P, De Sanjosé S, Bruni L. Worldwide use of HPV self-sampling for cervical cancer screening. Prev Med. 2022;154:106900.

Article   PubMed   CAS   Google Scholar  

Bray F, Ferlay J, Soerjomataram I, et al. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. Cancer J Clin. 2018;68(6):394–424.

Article   Google Scholar  

Arbyn M, Verdoodt F, Snijders PJ, Verhoef VM, Suonio E, Dillner L, Minozzi S, Bellisario C, Banzi R, Zhao FH, Hillemanns P. Accuracy of human papillomavirus testing on self-collected versus clinician-collected samples: a meta-analysis. Lancet Oncol. 2014;15(2):172–83.

Bosch FX, Broker TR, Forman D, et al. Comprehensive control of human papillomavirus infections and related diseases. Vaccine. 2013;31:H1–31.

Article   PubMed   PubMed Central   Google Scholar  

Arbyn M, Castle PE. Offering self-sampling devices for HPV testing to reach women who do not attend in the regular cervical cancer screening program. Cancer Epidemiol Biomarkers Prev. 2015;24(5):769–72.

Arbyn M, De Sanjosé S, Saraiya M, et al. EUROGIN 2011 roadmap on prevention and treatment of HPV-related disease. Int J Cancer. 2012;131(9):1969–82.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Arbyn M, Smith SB, Temin S et al. Detecting cervical precancer and reaching underscreened women by using HPV testing on self samples: updated meta-analyses. BMJ. 2018;363.

Snijders PJF, Verhoef VMJ, Arbyn M, et al. High-risk HPV testing on self‐sampled versus clinician‐collected specimens: a review on the clinical accuracy and impact on population attendance in cervical cancer screening. Int J Cancer. 2013;132(10):2223–36.

Sabeena S, Ravishankar N, Kalpana MM. Implementation strategies of cervical cancer screening in South Asia: a systematic review. Int J Gynecol Obstet. 2024.

Wong EL, Cheung AW, Wong AY, Chan PK. Acceptability and feasibility of HPV self-sampling as an alternative primary cervical cancer screening in under-screened population groups: a cross-sectional study. Int J Environ Res Public Health. 2020;17(17):6245.

Aasbø G, Tropè A, Nygård M, et al. HPV self-sampling among long-term nonattenders to cervical cancer screening in Norway: a pragmatic randomized controlled trial. Br J Cancer. 2022;127(10):1816–26.

Elfström KM, Sundström K, Andersson S, et al. Increasing participation in cervical screening by targeting long-term nonattenders: randomized health services study. Int J Cancer. 2019;145(11):3033–9.

Revised Cochrane risk-of-bias tool for randomized trials. RoB2 Development Group; 2019 [cited 2023 Jan 20]. https://drive.google.com/file/d/19R9savfPdCHC8XLz2iiMvL_71lPJERWK/view

Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Int J Surg. 2021;88:105906.

Arrossi S, Thouyaret L, Herrero R, et al. Effect of self-collection of HPV DNA offered by community health workers at home visits on uptake of screening for cervical cancer (the EMA study): a population-based cluster-randomized trial. Lancet Global Health. 2015;3(2):e85–94.

Enerly E, Bonde J, Schee K, et al. Self-sampling for human papillomavirus testing among nonattenders increases attendance to the Norwegian cervical cancer screening programme. PLoS ONE. 2016;11(4):e0151978.

Gizaw M, Teka B, Ruddies F, et al. Uptake of cervical cancer screening in Ethiopia by self-sampling HPV DNA compared to visual inspection with acetic acid: a cluster randomized trial. Cancer Prev Res. 2019;12(9):609–16.

Landy R, Hollingworth T, Waller J, et al. Nonspeculum sampling approaches for cervical screening in older women: randomized controlled trial. Br J Gen Pract. 2022;72(714):e26–33.

Modibbo F, Iregbu KC, Okuma J, et al. Randomized trial evaluating self-sampling for HPV DNA based tests for cervical cancer screening in Nigeria. Infect Agents cancer. 2017;12:1–9.

Moses E, Pedersen HN, Mitchell SM, et al. Uptake of community-based, self‐collected HPV testing vs. visual inspection with acetic acid for cervical cancer screening in Kampala, Uganda: preliminary results of a randomized controlled trial. Tropical Med Int Health. 2015;20(10):1355–67.

Article   CAS   Google Scholar  

Racey CS, Gesink DC, Burchell AN, et al. Randomized intervention of self-collected sampling for human papillomavirus testing in underscreened rural women: uptake of screening and acceptability. J Women’s Health. 2016;25(5):489–97.

Sancho-Garnier H, Tamalet C, Halfon P, et al. HPV self‐sampling or the Pap‐smear: a randomized study among cervical screening nonattenders from lower socioeconomic groups in France. Int J Cancer. 2013;133(11):2681–7.

PubMed   CAS   Google Scholar  

Sultana F, English DR, Simpson JA, et al. Home-based HPV self‐sampling improves participation by never‐screened and under‐screened women: results from a large randomized trial (iPap) in Australia. Int J Cancer. 2016;139(2):281–90.

Tranberg M, Bech BH, Blaakær J, et al. Preventing cervical cancer using HPV self-sampling: direct mailing of test-devices increases screening participation more than timely opt-in procedures-a randomized controlled trial. BMC Cancer. 2018;18:1–11.

Viviano M, Catarino R, Jeannot E, et al. Self-sampling to improve cervical cancer screening coverage in Switzerland: a randomized controlled trial. Br J Cancer. 2017;116(11):1382–8.

Winer RL, Lin J, Tiro JA, et al. Effect of mailed human papillomavirus test devices vs usual care reminders on cervical cancer screening uptake, precancer detection, and treatment: a randomized clinical trial. JAMA Netw open. 2019;2(11):e1914729.

Zehbe I, Jackson R, Wood B, et al. Community-randomized controlled trial embedded in the Anishinaabek cervical cancer screening study: human papillomavirus self-sampling versus papanicolaou cytology. BMJ open. 2016;6(10):e011754.

World Bank Country and lending groups. The Word Bank; [cited 2023 Jun 19]. https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups

Global strategy to accelerate the elimination of cervical cancer as a public health problem. [cited 2023 Jun 10]. https://www.who.int/publications-detail-redirect/9789240014107

Verdoodt F, Jentschke M, Hillemanns P, et al. Reaching women who do not participate in the regular cervical cancer screening programme by offering self-sampling devices: a systematic review and meta-analysis of randomized trials. Eur J Cancer. 2015;51(16):2375–85.

Yeh PT, Kennedy CE, De Vuyst H, et al. Self-sampling for human papillomavirus (HPV) testing: a systematic review and meta-analysis. BMJ Global Health. 2019;4(3):e001351.

Costa S, Verberckmoes B, Castle PE, Arbyn M. Offering HPV self-sampling kits: an updated meta-analysis of the effectiveness of strategies to increase participation in cervical cancer screening. Br J Cancer. 2023;128(5):805–13.

Download references

Acknowledgements

Authors would like to express our sincere thanks to Dr. Ahmat Ricky for providing his valuable opinion on conceptualization as well as his assistance in screening citations.

The authors received no financial support for the research, authorship, or publication of this article.

Author information

Authors and affiliations.

The Nethersole School of Nursing, Faculty of Medicine, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong, China

Ho Yan Wong

Centre for Health Systems and Policy Research, JC School of Public Health and Primary Care, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong SAR, China

Eliza Lai-yi Wong

JC School of Public Health and Primary Care, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong SAR, China

Rm 418, School of Public Health Building, Prince of Wales Hospital, Shatin, New Territories, Hong Kong SAR, China

You can also search for this author in PubMed   Google Scholar

Contributions

H.-Y.W. wrote the main manuscript text and prepared Tables  1 and 2 ; Figs.  1 , 2 , 3 , 4 , 5 and 6 . L.-Y. W. is responsible for reviewing and editing the main manuscript text. All authors reviewed the manuscript.

Corresponding author

Correspondence to Eliza Lai-yi Wong .

Ethics declarations

Ethics approval and consent to participate.

This study protocol was established in accordance with the Declaration of Helsinki.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Wong, H.Y., Wong, E.Ly. Invitation strategy of vaginal HPV self-sampling to improve participation in cervical cancer screening: a systematic review and meta-analysis of randomized trials. BMC Public Health 24 , 2461 (2024). https://doi.org/10.1186/s12889-024-19881-0

Download citation

Received : 25 June 2024

Accepted : 26 August 2024

Published : 10 September 2024

DOI : https://doi.org/10.1186/s12889-024-19881-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Cervical cancer
• Self-sampling
• Invitation strategy

BMC Public Health

ISSN: 1471-2458

• General enquiries: [email protected]