summative evaluation research design

Skip navigation

World Leaders in Research-Based User Experience

Formative vs. summative evaluations.

July 28, 2019 2019-07-28

• Email article
• Share on LinkedIn
• Share on Twitter

In the user-experience profession, we preach iteration and evaluation. There are two types of evaluation, formative and summative, and where you are in the design process determines what type of evaluation you should conduct.

Formative evaluations focus on determining which aspects of the design work well or not, and why. These evaluations occur throughout a redesign and provide information to incrementally improve the interface.

Let’s say we’re designing the onboarding experience for a new, completely redesigned version of our mobile app. In the design process, we prototype a solution and then test it with ( usually a few ) users to see how usable it is. The study identifies several issues with our prototype, which are then fixed by a new design. This test is an example of formative evaluation — it helps designers identify what needs to be changed to improve the interface.

Formative evaluations of interfaces involve testing and changing the product, usually multiple times, and therefore are well-suited for the redesign process or while creating a new product.

In both cases, you iterate through the prototyping and testing steps until you are as ready for production as you’ll get (even more iterations would form an even better design, but you have to ship at some point). Thus, formative evaluations are meant to steer the design on the right path.

Summative evaluations describe how well a design performs , often compared to a benchmark such as a prior version of the design or a competitor. Unlike formative evaluations, whose goals is to inform the design process, summative evaluations involve getting the big picture and assessing the overall experience of a finished product. Summative evaluations occur less frequently than formative evaluations, usually right before or right after a redesign.

Let’s go back to our mobile-app example. Now that we’ve shipped the new mobile app, it is time to run a study and see how our app stands in comparison to the previous version of the app. We can gather the time on task and the success rates for the core app functionalities. Then we can compare these metrics against those obtained with the previous version of the app to see if there was any improvement. We will also save the results of this study to evaluate subsequent major versions of the app. This type of study is a summative evaluation since it assesses the shipped product with the goal of tracking performance over time and ultimately calculating our return on investment . However, during this study, we might uncover some usability issues. We should make note of those issues and address them during our next design iteration.

Alternatively, another type of summative evaluations could compare our results with those obtained with one or more competitor apps or with known industry-wide data.

All summative evaluations paint an overview picture of the usability of a system. They are intended to serve as reference points so that you can determine whether you’re improving your own designs over time or beating out a competitor.

The ultimate summative evaluation is the go/no-go decision of whether to release a product. After all is said and done, is your design good enough to be inflicted on the public, or do we think that it will harm our brand so badly that it should never see the light of day? It’s actually rare for companies to have a formal process to kill off bad design, which may be why we encounter many releases that do more harm than good for a brand. If you truly embrace our proposition that brand is experience in the digital age, then consider a final summative evaluation before release.

In This Article:

Origin of the terms, when each type of evaluation is used, research methods for formative vs. summative evaluations.

The terms ‘formative’ and ‘summative’ evaluation were coined by Michael Scriven in 1967. These terms were presented in the context of instructional design and education theory, but are just as valuable for any sort of evaluation-based industry.

In the educational context, formative evaluations are ongoing and occur throughout the development of the course, while summative evaluations occur less frequently and are used to determine whether the program met its intended goals. The formative evaluations are used to steer the teaching, by testing whether content was understood or needs to be revisited, while summative evaluations assess the student’s mastery of the material.

Recall that formative and summative evaluations align with your place in the design process. Formative evaluations go with prototype and testing iterations throughout a redesign project, while summative evaluations are best for right before or right after a major redesign.

Great researchers begin their study by determining what question they’re trying to answer. Essentially, your research question is the same as the type of evaluation. Below is a list of possible research questions you might have and the corresponding evaluation. For that reason, this table is descriptive, not prescriptive.

After it is clear which type of evaluation you will conduct, you have to determine which research method you should use. There is a common misconception that summative equals quantitative and formative equals qualitative ­­— this is not the case.  

Summative evaluations can be either qualitative or quantitative. The same is true for formative evaluations.

Although summative evaluations are often quantitative, they can be qualitative studies, too. For example, you might like to know where your product stands compared with your competition. You could hire a UX expert to do an expert review of your interface and a competitor’s. The expert review would use the 10 usability heuristics as well as the reviewer’s knowledge of UI and human behavior to produce a list of strength and weaknesses for both your interface and your competitor’s. The study is summative because the overall interface is being evaluated with the goal of understanding whether the UX of your product stands up to the competition and whether a major redesign is warranted.

Additionally, formative evaluations aren’t always qualitative, although that is often the case. (Since it’s recommended to run an extended series of formative evaluations, it makes financial sense to use a cheaper qualitative study for each of them.) But sometimes big companies with large UX budgets and high level of UX maturity  might use quantitative studies for formative purposes in order to ensure that a change to one of their essential features will perform satisfactorily.  For instance, before launching a new homepage design, a large company may want to run a quantitative test on the prototype to make sure that the number of people who will scroll below the fold is high enough. 

Formative and summative evaluations correspond to different research goals. Formative evaluations are meant to steer the design on the correct path so that the final product has satisfactory user experience. They are a natural part of any iterative user-centered design process. Summative evaluations assess the overall usability of a product and are instrumental in tracking its usability over time and in comparing it with competitors.

Greenstein, Laura.  What Teachers Really Need to Know About Formative Assessment . ASCD, 2010.

Related Courses

Discovery: building the right thing.

Conduct successful discovery phases to ensure you build the best solution

User Research Methods: From Strategy to Requirements to Design

Pick the best UX research method for each stage in the design process

ResearchOps: Scaling User Research

Orchestrate and optimize research to amplify its impact

Related Topics

• Research Methods Research Methods

Learn More:

Please accept marketing cookies to view the embedded video. https://www.youtube.com/watch?v=730UiP7dZeo

Formative vs. Summative Usability Evaluation

Always Pilot Test User Research Studies

Kim Salazar · 3 min

Level Up Your Focus Groups

Therese Fessenden · 5 min

Inductively Analyzing Qualitative Data

Tanner Kohler · 3 min

Related Articles:

Open-Ended vs. Closed Questions in User Research

Maria Rosala · 5 min

Cognitive Mapping in User Research

Sarah Gibbons · 16 min

UX Research Methods: Glossary

Raluca Budiu · 12 min

Iterative Design of a Survey Question: A Case Study

Feifei Liu · 8 min

When to Use Which User-Experience Research Methods

Christian Rohrer · 9 min

You Are Not the User: The False-Consensus Effect

Raluca Budiu · 4 min

• Evaluation Research Design: Examples, Methods & Types

As you engage in tasks, you will need to take intermittent breaks to determine how much progress has been made and if any changes need to be effected along the way. This is very similar to what organizations do when they carry out  evaluation research.  

The evaluation research methodology has become one of the most important approaches for organizations as they strive to create products, services, and processes that speak to the needs of target users. In this article, we will show you how your organization can conduct successful evaluation research using Formplus .

What is Evaluation Research?

Also known as program evaluation, evaluation research is a common research design that entails carrying out a structured assessment of the value of resources committed to a project or specific goal. It often adopts social research methods to gather and analyze useful information about organizational processes and products.  

As a type of applied research , evaluation research typically associated  with real-life scenarios within organizational contexts. This means that the researcher will need to leverage common workplace skills including interpersonal skills and team play to arrive at objective research findings that will be useful to stakeholders. 

Characteristics of Evaluation Research

• Research Environment: Evaluation research is conducted in the real world; that is, within the context of an organization. 
• Research Focus: Evaluation research is primarily concerned with measuring the outcomes of a process rather than the process itself. 
• Research Outcome: Evaluation research is employed for strategic decision making in organizations. 
• Research Goal: The goal of program evaluation is to determine whether a process has yielded the desired result(s). 
• This type of research protects the interests of stakeholders in the organization. 
• It often represents a middle-ground between pure and applied research. 
• Evaluation research is both detailed and continuous. It pays attention to performative processes rather than descriptions. 
• Research Process: This research design utilizes qualitative and quantitative research methods to gather relevant data about a product or action-based strategy. These methods include observation, tests, and surveys.

Types of Evaluation Research

The Encyclopedia of Evaluation (Mathison, 2004) treats forty-two different evaluation approaches and models ranging from “appreciative inquiry” to “connoisseurship” to “transformative evaluation”. Common types of evaluation research include the following: 

• Formative Evaluation

Formative evaluation or baseline survey is a type of evaluation research that involves assessing the needs of the users or target market before embarking on a project.  Formative evaluation is the starting point of evaluation research because it sets the tone of the organization’s project and provides useful insights for other types of evaluation.  

• Mid-term Evaluation

Mid-term evaluation entails assessing how far a project has come and determining if it is in line with the set goals and objectives. Mid-term reviews allow the organization to determine if a change or modification of the implementation strategy is necessary, and it also serves for tracking the project. 

• Summative Evaluation

This type of evaluation is also known as end-term evaluation of project-completion evaluation and it is conducted immediately after the completion of a project. Here, the researcher examines the value and outputs of the program within the context of the projected results. 

Summative evaluation allows the organization to measure the degree of success of a project. Such results can be shared with stakeholders, target markets, and prospective investors. 

• Outcome Evaluation

Outcome evaluation is primarily target-audience oriented because it measures the effects of the project, program, or product on the users. This type of evaluation views the outcomes of the project through the lens of the target audience and it often measures changes such as knowledge-improvement, skill acquisition, and increased job efficiency. 

• Appreciative Enquiry

Appreciative inquiry is a type of evaluation research that pays attention to result-producing approaches. It is predicated on the belief that an organization will grow in whatever direction its stakeholders pay primary attention to such that if all the attention is focused on problems, identifying them would be easy. 

In carrying out appreciative inquiry, the research identifies the factors directly responsible for the positive results realized in the course of a project, analyses the reasons for these results, and intensifies the utilization of these factors. 

Evaluation Research Methodology 

There are four major evaluation research methods, namely; output measurement, input measurement, impact assessment and service quality

• Output/Performance Measurement

Output measurement is a method employed in evaluative research that shows the results of an activity undertaking by an organization. In other words, performance measurement pays attention to the results achieved by the resources invested in a specific activity or organizational process. 

More than investing resources in a project, organizations must be able to track the extent to which these resources have yielded results, and this is where performance measurement comes in. Output measurement allows organizations to pay attention to the effectiveness and impact of a process rather than just the process itself. 

Other key indicators of performance measurement include user-satisfaction, organizational capacity, market penetration, and facility utilization. In carrying out performance measurement, organizations must identify the parameters that are relevant to the process in question, their industry, and the target markets. 

5 Performance Evaluation Research Questions Examples

• What is the cost-effectiveness of this project?
• What is the overall reach of this project?
• How would you rate the market penetration of this project?
• How accessible is the project? 
• Is this project time-efficient? 

• Input Measurement

In evaluation research, input measurement entails assessing the number of resources committed to a project or goal in any organization. This is one of the most common indicators in evaluation research because it allows organizations to track their investments. 

The most common indicator of inputs measurement is the budget which allows organizations to evaluate and limit expenditure for a project. It is also important to measure non-monetary investments like human capital; that is the number of persons needed for successful project execution and production capital. 

5 Input Evaluation Research Questions Examples

• What is the budget for this project?
• What is the timeline of this process?
• How many employees have been assigned to this project? 
• Do we need to purchase new machinery for this project? 
• How many third-parties are collaborators in this project? 

• Impact/Outcomes Assessment

In impact assessment, the evaluation researcher focuses on how the product or project affects target markets, both directly and indirectly. Outcomes assessment is somewhat challenging because many times, it is difficult to measure the real-time value and benefits of a project for the users. 

In assessing the impact of a process, the evaluation researcher must pay attention to the improvement recorded by the users as a result of the process or project in question. Hence, it makes sense to focus on cognitive and affective changes, expectation-satisfaction, and similar accomplishments of the users. 

5 Impact Evaluation Research Questions Examples

• How has this project affected you? 
• Has this process affected you positively or negatively?
• What role did this project play in improving your earning power? 
• On a scale of 1-10, how excited are you about this project?
• How has this project improved your mental health? 

• Service Quality

Service quality is the evaluation research method that accounts for any differences between the expectations of the target markets and their impression of the undertaken project. Hence, it pays attention to the overall service quality assessment carried out by the users. 

It is not uncommon for organizations to build the expectations of target markets as they embark on specific projects. Service quality evaluation allows these organizations to track the extent to which the actual product or service delivery fulfils the expectations. 

5 Service Quality Evaluation Questions

• On a scale of 1-10, how satisfied are you with the product?
• How helpful was our customer service representative?
• How satisfied are you with the quality of service?
• How long did it take to resolve the issue at hand?
• How likely are you to recommend us to your network?

Uses of Evaluation Research 

• Evaluation research is used by organizations to measure the effectiveness of activities and identify areas needing improvement. Findings from evaluation research are key to project and product advancements and are very influential in helping organizations realize their goals efficiently.     
• The findings arrived at from evaluation research serve as evidence of the impact of the project embarked on by an organization. This information can be presented to stakeholders, customers, and can also help your organization secure investments for future projects. 
• Evaluation research helps organizations to justify their use of limited resources and choose the best alternatives. 
•  It is also useful in pragmatic goal setting and realization. 
• Evaluation research provides detailed insights into projects embarked on by an organization. Essentially, it allows all stakeholders to understand multiple dimensions of a process, and to determine strengths and weaknesses. 
• Evaluation research also plays a major role in helping organizations to improve their overall practice and service delivery. This research design allows organizations to weigh existing processes through feedback provided by stakeholders, and this informs better decision making. 
• Evaluation research is also instrumental to sustainable capacity building. It helps you to analyze demand patterns and determine whether your organization requires more funds, upskilling or improved operations.

Data Collection Techniques Used in Evaluation Research

In gathering useful data for evaluation research, the researcher often combines quantitative and qualitative research methods . Qualitative research methods allow the researcher to gather information relating to intangible values such as market satisfaction and perception. 

On the other hand, quantitative methods are used by the evaluation researcher to assess numerical patterns, that is, quantifiable data. These methods help you measure impact and results; although they may not serve for understanding the context of the process. 

Quantitative Methods for Evaluation Research

A survey is a quantitative method that allows you to gather information about a project from a specific group of people. Surveys are largely context-based and limited to target groups who are asked a set of structured questions in line with the predetermined context.

Surveys usually consist of close-ended questions that allow the evaluative researcher to gain insight into several  variables including market coverage and customer preferences. Surveys can be carried out physically using paper forms or online through data-gathering platforms like Formplus . 

• Questionnaires

A questionnaire is a common quantitative research instrument deployed in evaluation research. Typically, it is an aggregation of different types of questions or prompts which help the researcher to obtain valuable information from respondents. 

A poll is a common method of opinion-sampling that allows you to weigh the perception of the public about issues that affect them. The best way to achieve accuracy in polling is by conducting them online using platforms like Formplus. 

Polls are often structured as Likert questions and the options provided always account for neutrality or indecision. Conducting a poll allows the evaluation researcher to understand the extent to which the product or service satisfies the needs of the users. 

Qualitative Methods for Evaluation Research

• One-on-One Interview

An interview is a structured conversation involving two participants; usually the researcher and the user or a member of the target market. One-on-One interviews can be conducted physically, via the telephone and through video conferencing apps like Zoom and Google Meet. 

• Focus Groups

A focus group is a research method that involves interacting with a limited number of persons within your target market, who can provide insights on market perceptions and new products. 

• Qualitative Observation

Qualitative observation is a research method that allows the evaluation researcher to gather useful information from the target audience through a variety of subjective approaches. This method is more extensive than quantitative observation because it deals with a smaller sample size, and it also utilizes inductive analysis. 

• Case Studies

A case study is a research method that helps the researcher to gain a better understanding of a subject or process. Case studies involve in-depth research into a given subject, to understand its functionalities and successes. 

How to Formplus Online Form Builder for Evaluation Survey 

• Sign into Formplus

In the Formplus builder, you can easily create your evaluation survey by dragging and dropping preferred fields into your form. To access the Formplus builder, you will need to create an account on Formplus. 

Once you do this, sign in to your account and click on “Create Form ” to begin. 

• Edit Form Title

Click on the field provided to input your form title, for example, “Evaluation Research Survey”.

Click on the edit button to edit the form.

Add Fields: Drag and drop preferred form fields into your form in the Formplus builder inputs column. There are several field input options for surveys in the Formplus builder. 

Edit fields

Click on “Save”

Preview form.

• Form Customization

With the form customization options in the form builder, you can easily change the outlook of your form and make it more unique and personalized. Formplus allows you to change your form theme, add background images, and even change the font according to your needs. 

• Multiple Sharing Options

Formplus offers multiple form sharing options which enables you to easily share your evaluation survey with survey respondents. You can use the direct social media sharing buttons to share your form link to your organization’s social media pages. 

You can send out your survey form as email invitations to your research subjects too. If you wish, you can share your form’s QR code or embed it on your organization’s website for easy access. 

Conclusion  

Conducting evaluation research allows organizations to determine the effectiveness of their activities at different phases. This type of research can be carried out using qualitative and quantitative data collection methods including focus groups, observation, telephone and one-on-one interviews, and surveys. 

Online surveys created and administered via data collection platforms like Formplus make it easier for you to gather and process information during evaluation research. With Formplus multiple form sharing options, it is even easier for you to gather useful data from target markets.

Connect to Formplus, Get Started Now - It's Free!

• characteristics of evaluation research
• evaluation research methods
• types of evaluation research
• what is evaluation research
• busayo.longe

You may also like:

Assessment vs Evaluation: 11 Key Differences

This article will discuss what constitutes evaluations and assessments along with the key differences between these two research methods.

What is Pure or Basic Research? + [Examples & Method]

Simple guide on pure or basic research, its methods, characteristics, advantages, and examples in science, medicine, education and psychology

Formal Assessment: Definition, Types Examples & Benefits

In this article, we will discuss different types and examples of formal evaluation, and show you how to use Formplus for online assessments.

Recall Bias: Definition, Types, Examples & Mitigation

This article will discuss the impact of recall bias in studies and the best ways to avoid them during research.

Formplus - For Seamless Data Collection

Collect data the right way with a versatile data collection tool. try formplus and transform your work productivity today..

• Understanding Summative Evaluation: Definition, Benefits, and Best Practices
• Learning Center

This article provides an overview of summative evaluation, including its definition, benefits, and best practices. Discover how summative evaluation can help you assess the effectiveness of your program or project, identify areas for improvement, and promote evidence-based decision-making. Learn about best practices for conducting summative evaluation and how to address common challenges and limitations.

Table of Contents

What is Summative Evaluation and Why is it Important?

Summative evaluation: purpose, goals, benefits of summative evaluation, types of summative evaluation, best practices for conducting summative evaluation, examples of summative evaluation in practice, examples of summative evaluation questions, challenges and limitations of summative evaluation, ensuring ethical considerations in summative evaluation, future directions for summative evaluation research and practice.

Ready to optimize your resume for a better career? Try Our FREE Resume Scanner!

Optimize your resume for ATS with formatting, keywords, and quantified experience.

• Compare job description keywords with your resume.
• Tailor your resume to match as many keywords as possible.
• Grab the recruiter’s attention with a standout resume.

Summative evaluation is a type of evaluation that is conducted at the end of a program or project, with the goal of assessing its overall effectiveness. The primary focus of summative evaluation is to determine whether the program or project achieved its goals and objectives. Summative evaluation is often used to inform decisions about future program or project development, as well as to determine whether or not to continue funding a particular program or project.

Summative evaluation is important for several reasons. First, it provides a comprehensive assessment of the overall effectiveness of a program or project, which can help to inform decisions about future development and implementation. Second, it can help to identify areas where improvements can be made in program delivery, such as in program design or implementation. Third, it can help to determine whether the program or project is a worthwhile investment, and whether it is meeting the needs of stakeholders.

In addition to these benefits, summative evaluation can also help to promote accountability and transparency in program or project implementation. By conducting a thorough evaluation of the program or project, stakeholders can be assured that their resources are being used effectively and that the program or project is achieving its intended outcomes.

Summative evaluation plays an important role in assessing the overall effectiveness of a program or project, and in informing decisions about future development and implementation. It is an essential tool for promoting accountability, transparency, and effectiveness in program or project implementation.

Summative evaluation is an approach to program evaluation that is conducted at the end of a program or project, with the goal of assessing its overall effectiveness. Here are some of the key purposes and goals of summative evaluation.

Purpose of Summative Evaluation

• Assess effectiveness: Summative evaluation is focused on assessing the overall effectiveness of a program or project in achieving its intended goals and objectives.
• Determine impact: Summative evaluation is used to determine the impact of a program or project on its intended audience or stakeholders, as well as on the broader community or environment.
• Inform decision-making: Summative evaluation is used to inform decision-making about future program or project development, as well as resource allocation.

Goals of Summative Evaluation

• Measure program outcomes: Summative evaluation is used to measure program outcomes, including the extent to which the program achieved its intended goals and objectives, and the impact of the program on its intended audience or stakeholders.
• Assess program effectiveness: Summative evaluation is used to assess the overall effectiveness of a program, by comparing program outcomes to its intended goals and objectives, as well as to similar programs or initiatives.
• Inform program improvement: Summative evaluation is used to inform program improvement by identifying areas where the program could be modified or improved in order to enhance its effectiveness.

Summative evaluation is a critical tool for assessing the overall effectiveness and impact of programs or projects, and for informing decision-making about future program or project development. By measuring program outcomes, assessing program effectiveness, and identifying areas for program improvement, summative evaluation can help to ensure that programs and projects are meeting their intended goals and making a positive impact on their intended audience or stakeholders.

Summative evaluation is an important tool for assessing the overall effectiveness of a program or project. Here are some of the benefits of conducting summative evaluation:

• Provides a Comprehensive Assessment: Summative evaluation provides a comprehensive assessment of the overall effectiveness of a program or project, which can help to inform decisions about future development and implementation.
• Identifies Areas for Improvement : Summative evaluation can help to identify areas where improvements can be made in program delivery, such as in program design or implementation.
• Promotes Accountability and Transparency: Summative evaluation can help to promote accountability and transparency in program or project implementation, by ensuring that resources are being used effectively and that the program or project is achieving its intended outcomes.
• Supports Evidence-Based Decision-Making : Summative evaluation provides evidence-based data and insights that can inform decisions about future development and implementation.
• Demonstrates Impact : Summative evaluation can help to demonstrate the impact of a program or project, which can be useful for securing funding or support for future initiatives.
• Increases Stakeholder Engagement : Summative evaluation can increase stakeholder engagement and ownership of the program or project being evaluated, by involving stakeholders in the evaluation process and soliciting their feedback.

Summative evaluation is an essential tool for assessing the overall effectiveness of a program or project, and for informing decisions about future development and implementation. It provides a comprehensive assessment of the program or project, identifies areas for improvement, promotes accountability and transparency, and supports evidence-based decision-making.

There are different types of summative evaluation that can be used to assess the overall effectiveness of a program or project. Here are some of the most common types of summative evaluation:

• Outcome Evaluation: This type of evaluation focuses on the outcomes or results of the program or project, such as changes in behavior, knowledge, or attitudes. Outcome evaluation is often used to determine the effectiveness of an intervention or program in achieving its intended outcomes.
• Impact Evaluation: This type of evaluation focuses on the broader impact of the program or project, such as changes in the community or society. Impact evaluation is often used to assess the overall impact of a program or project on the target population or community.
• Cost-Benefit Evaluation: This type of evaluation focuses on the costs and benefits of the program or project, and is often used to determine whether the program or project is a worthwhile investment. Cost-benefit evaluation can help to determine whether the benefits of the program or project outweigh the costs.

The type of summative evaluation used will depend on the specific goals and objectives of the program or project being evaluated, as well as the resources and data available for evaluation. Each type of summative evaluation serves a specific purpose in assessing the overall effectiveness of a program or project, and should be tailored to the specific needs of the program or project being evaluated.

Conducting a successful summative evaluation requires careful planning and attention to best practices. Here are some best practices for conducting summative evaluation:

• Clearly Define Goals and Objectives : Before conducting a summative evaluation, it is important to clearly define the goals and objectives of the program or project being evaluated. This will help to ensure that the evaluation is focused and relevant to the needs of stakeholders.
• Use Valid and Reliable Measures: The measures used in a summative evaluation should be valid and reliable, in order to ensure that the results are accurate and meaningful. This may involve selecting or developing appropriate evaluation tools, such as surveys or assessments, and ensuring that they are properly administered.
• Collect Data from Multiple Sources : Data for a summative evaluation should be collected from multiple sources, in order to ensure that the results are comprehensive and representative. This may involve collecting data from program participants, stakeholders, and other relevant sources.
• Analyze and Interpret Results : Once the data has been collected, it is important to analyze and interpret the results in order to determine the overall effectiveness of the program or project. This may involve using statistical analysis or other techniques to identify patterns or trends in the data.
• Use Results to Inform Future Development : The results of a summative evaluation should be used to inform future program or project development, in order to improve the effectiveness of the program or project. This may involve making changes to program design or delivery, or identifying areas where additional resources or support may be needed.

Conducting a successful summative evaluation requires careful planning, attention to detail, and a commitment to using the results to inform future development and improvement. By following best practices for conducting summative evaluation, stakeholders can ensure that their programs and projects are effective and relevant to the needs of their communities.

Summative evaluation is an important tool for assessing the overall effectiveness of a program or project. Here are some examples of summative evaluation in practice:

• Educational Programs : A school district may conduct a summative evaluation of a new educational program, such as a reading intervention program. The evaluation may focus on the program’s outcomes, such as improvements in reading skills, and may involve collecting data from multiple sources, such as teacher assessments, student tests, and parent surveys.
• Health Interventions : A public health agency may conduct a summative evaluation of a health intervention, such as a vaccination campaign. The evaluation may focus on the impact of the intervention on health outcomes, such as reductions in disease incidence, and may involve collecting data from multiple sources, such as healthcare providers, patients, and community members.
• Social Service Programs: A non-profit organization may conduct a summative evaluation of a social service program, such as a job training program for disadvantaged youth. The evaluation may focus on the impact of the program on outcomes such as employment rates and job retention, and may involve collecting data from multiple sources, such as program participants, employers, and community partners.
• Technology Products : A software company may conduct a summative evaluation of a new technology product, such as a mobile app. The evaluation may focus on user satisfaction and effectiveness, and may involve collecting data from multiple sources, such as user surveys, user testing, and usage data.
• Environmental Programs : An environmental organization may conduct a summative evaluation of a conservation program, such as a land protection initiative. The evaluation may focus on the impact of the program on environmental outcomes, such as the protection of natural habitats or the reduction of greenhouse gas emissions, and may involve collecting data from multiple sources, such as program participants, community members, and scientific data.

Summative evaluation can be used in a wide range of programs and initiatives to assess their overall effectiveness and inform future development and improvement.

Summative evaluation is an important tool for assessing the overall effectiveness of a program or project. Here are some examples of summative evaluation questions that can be used to guide the evaluation process:

• Did the program or project achieve its intended outcomes and goals?
• To what extent did the program or project meet the needs of its intended audience or stakeholders?
• What were the most effective components of the program or project, and what areas could be improved?
• What impact did the program or project have on its intended audience or stakeholders?
• Was the program or project implemented effectively, and were resources used efficiently?
• What unintended consequences or challenges arose during the program or project, and how were they addressed?
• How does the program or project compare to similar initiatives or interventions in terms of effectiveness and impact?
• What were the costs and benefits of the program or project, and were they reasonable given the outcomes achieved?
• What lessons can be learned from the program or project, and how can they inform future development and improvement?

The questions asked during a summative evaluation are designed to provide a comprehensive understanding of the impact and effectiveness of the program or project. The answers to these questions can inform future programming and resource allocation decisions and help to identify areas for improvement. Overall, summative evaluation is an essential tool for assessing the overall impact and effectiveness of a program or project.

Summative evaluation is an important tool for assessing the overall effectiveness of a program or project. However, there are several challenges and limitations that should be considered when conducting summative evaluation. Here are some of the most common challenges and limitations of summative evaluation:

• Timing: Summative evaluation is typically conducted at the end of a program or project, which may limit the ability to make real-time improvements during the implementation phase.
• Resource Constraints: Summative evaluation can be resource-intensive, requiring significant time, effort, and funding to collect and analyze data.
• Bias: The data collected during summative evaluation may be subject to bias, such as social desirability bias, which can affect the accuracy and reliability of the evaluation results.
• Difficulty of Measurement: Some outcomes of a program or project may be difficult to measure, which can make it challenging to assess the overall effectiveness of the program or project.
• Difficulty of Generalization: The results of a summative evaluation may not be generalizable to other contexts or settings, which can limit the broader applicability of the evaluation findings.
• Limited Stakeholder Involvement: Summative evaluation may not involve all stakeholders, which can limit the representation of diverse perspectives and lead to incomplete evaluation findings.
• Limited Focus on Process: Summative evaluation typically focuses on outcomes and impact, which may not provide a full understanding of the program or project’s implementation process and effectiveness.

These challenges and limitations of summative evaluation should be considered when planning and conducting evaluations. By understanding these limitations, evaluators can work to mitigate potential biases and limitations and ensure that the evaluation results are accurate, reliable, and useful for program or project improvement.

While conducting summative evaluation, it’s imperative to uphold ethical principles to ensure the integrity and fairness of the evaluation process. Ethical considerations are essential for maintaining trust with stakeholders, respecting the rights of participants, and safeguarding the integrity of evaluation findings. Here are key ethical considerations to integrate into summative evaluation:

Informed Consent: Ensure that participants are fully informed about the purpose, procedures, risks, and benefits of the evaluation before consenting to participate. Provide clear and accessible information, allowing participants to make voluntary and informed decisions about their involvement.

Confidentiality and Privacy: Safeguard the confidentiality and privacy of participants’ information throughout the evaluation process. Implement secure data management practices, anonymize data whenever possible, and only share findings in aggregate or de-identified formats to protect participants’ identities.

Respect for Diversity and Inclusion: Respect and embrace the diversity of participants, acknowledging their unique perspectives, backgrounds, and experiences. Ensure that evaluation methods are culturally sensitive and inclusive, avoiding biases and stereotypes, and accommodating diverse communication styles and preferences.

Avoiding Harm: Take proactive measures to minimize the risk of harm to participants and stakeholders throughout the evaluation process. Anticipate potential risks and vulnerabilities, mitigate them through appropriate safeguards and protocols, and prioritize the well-being and dignity of all involved.

Beneficence and Non-Maleficence: Strive to maximize the benefits of the evaluation while minimizing any potential harm or adverse effects. Ensure that evaluation activities contribute to the improvement of programs or projects, enhance stakeholders’ understanding and decision-making, and do not cause undue stress, discomfort, or harm.

Transparency and Accountability: Maintain transparency and accountability in all aspects of the evaluation, including its design, implementation, analysis, and reporting. Clearly communicate the evaluation’s objectives, methodologies, findings, and limitations, allowing stakeholders to assess its credibility and relevance.

Equitable Participation and Representation: Foster equitable participation and representation of diverse stakeholders throughout the evaluation process. Engage stakeholders in meaningful ways, valuing their input, perspectives, and contributions, and address power differentials to ensure inclusive decision-making and ownership of evaluation outcomes.

Continuous Reflection and Improvement: Continuously reflect on ethical considerations throughout the evaluation process, remaining responsive to emerging issues, challenges, and ethical dilemmas. Seek feedback from stakeholders, engage in dialogue about ethical concerns, and adapt evaluation approaches accordingly to uphold ethical standards.

By integrating these ethical considerations into summative evaluation practices, evaluators can uphold principles of integrity, respect, fairness, and accountability, promoting trust, credibility, and meaningful impact in program assessment and improvement. Ethical evaluation practices not only ensure compliance with professional standards and legal requirements but also uphold fundamental values of respect for human dignity, justice, and social responsibility.

Summative evaluation is an important tool for assessing the overall effectiveness of a program or project. Here are some potential future directions for summative evaluation research and practice:

• Incorporating Technology: Advances in technology have the potential to improve the efficiency and accuracy of summative evaluation. Future research could explore the use of artificial intelligence, machine learning, and other technologies to streamline data collection and analysis.
• Enhancing Stakeholder Engagement: Future research could explore ways to enhance stakeholder engagement in summative evaluation, such as by involving stakeholders in the evaluation planning and implementation process.
• Increasing Use of Mixed Methods: Future research could explore the use of mixed methods approaches in summative evaluation, such as combining qualitative and quantitative methods to gain a more comprehensive understanding of program or project effectiveness.
• Addressing Equity and Inclusion: Future research could focus on addressing issues of equity and inclusion in summative evaluation, such as by ensuring that evaluation methods are sensitive to the needs and experiences of diverse stakeholders.
• Addressing Complexity: Many programs and projects operate in complex and dynamic environments. Future research could explore ways to address this complexity in summative evaluation, such as by developing more adaptive and flexible evaluation methods.
• Improving Integration with Formative Evaluation: Summative evaluation is typically conducted after a program or project has been completed, while formative evaluation is conducted during program or project implementation. Future research could explore ways to better integrate summative and formative evaluation, in order to promote continuous program improvement.

These future directions for summative evaluation research and practice have the potential to improve the effectiveness and relevance of summative evaluation, and to enhance its value as a tool for program and project assessment and improvement.

David Ndiyamba

Quite comprehensive Thank you

Fation Luli

Leave a comment cancel reply.

Your email address will not be published.

Login with your Social Account

How strong is my resume.

Only 2% of resumes land interviews.

Land a better, higher-paying career

Jobs for You

Business development associate.

• United States

Director of Finance and Administration

• Bosnia and Herzegovina

Request for Information – Collecting Information on Potential Partners for Local Works Evaluation

• Washington, USA

Principal Field Monitors

Technical expert (health, wash, nutrition, education, child protection, hiv/aids, supplies), survey expert, data analyst, team leader, usaid-bha performance evaluation consultant.

• International Rescue Committee

Manager II, Institutional Support Program Implementation

Senior human resources associate, energy and environment analyst – usaid bureau for latin america and the caribbean, intern- international project and proposal support, ispi, deputy chief of party, senior accounting associate, services you might be interested in, useful guides ....

How to Create a Strong Resume

Monitoring And Evaluation Specialist Resume

Resume Length for the International Development Sector

Types of Evaluation

Monitoring, Evaluation, Accountability, and Learning (MEAL)

LAND A JOB REFERRAL IN 2 WEEKS (NO ONLINE APPS!)

Sign Up & To Get My Free Referral Toolkit Now:

Eberly Center

Teaching excellence & educational innovation, what is the difference between formative and summative assessment, formative assessment.

The goal of formative assessment is to monitor student learning to provide ongoing feedback that can be used by instructors to improve their teaching and by students to improve their learning. More specifically, formative assessments:

• help students identify their strengths and weaknesses and target areas that need work
• help faculty recognize where students are struggling and address problems immediately

Formative assessments are generally low stakes , which means that they have low or no point value. Examples of formative assessments include asking students to:

• draw a concept map in class to represent their understanding of a topic
• submit one or two sentences identifying the main point of a lecture
• turn in a research proposal for early feedback

Summative assessment

The goal of summative assessment is to evaluate student learning at the end of an instructional unit by comparing it against some standard or benchmark.

Summative assessments are often high stakes , which means that they have a high point value. Examples of summative assessments include:

• a midterm exam
• a final project
• a senior recital

Information from summative assessments can be used formatively when students or faculty use it to guide their efforts and activities in subsequent courses.

CONTACT US to talk with an Eberly colleague in person!

• Faculty Support
• Graduate Student Support
• Canvas @ Carnegie Mellon
• Quick Links

Summative Assessment and Feedback

Main navigation.

Summative assessments are given to students at the end of a course and should measure the skills and knowledge a student has gained over the entire instructional period. Summative feedback is aimed at helping students understand how well they have done in meeting the overall learning goals of the course.

Effective summative assessments

Effective summative assessments provide students a structured way to demonstrate that they have met a range of key learning objectives and to receive useful feedback on their overall learning. They should align with the course learning goals and build upon prior formative assessments. These assessments will address how well the student is able to synthesize and connect the elements of learning from the entirety of the course into a holistic understanding and provide an opportunity to provide rich summative feedback.

The value of summative feedback

Summative feedback is essential for students to understand how far they have come in meeting the learning goals of the course, what they need further work on, and what they should study next. This can affect later choices that students make, particularly in contemplating and pursuing their major fields of study. Summative feedback can also influence how students regard themselves and their academic disciplines after graduation.

Use rubrics to provide consistency and transparency

A rubric is a grading guide for evaluating how well students have met a learning outcome. A rubric consists of performance criteria, a rating scale, and indicators for the different rating levels. They are typically in a chart or table format. 

Instructors often use rubrics for both formative and summative feedback to ensure consistency of assessment across different students. Rubrics also can make grading faster and help to create consistency between multiple graders and across assignments.

Students might be given access to the rubric before working on an assignment. No criteria or metric within a summative assessment should come as a surprise to the students. Transparency with students on exactly what is being assessed can help them more effectively demonstrate how much they have learned.  

Types of  summative assessments

Different summative assessments are better suited to measuring different kinds of learning. 

Examinations

Examinations are useful for evaluating student learning in terms of remembering information, and understanding and applying concepts and ideas. However, exams may be less suited to evaluating how well students are able to analyze, evaluate, or create things related to what they've learned.

Presentation

A presentation tasks the student with teaching others what they have learned typically by speaking, presenting visual materials, and interacting with their audience. This can be useful for assessing a student's ability to critically analyze and evaluate a topic or content.

With projects, students will create something, such as a plan, document, artifact, or object, usually over a sustained period of time, that demonstrates skills or understanding of the topic of learning. They are useful for evaluating learning objectives that require high levels of critical thinking, creativity, and coordination. Projects are good opportunities to provide summative feedback because they often build on prior formative assessments and feedback. 

With a portfolio, students create and curate a collection of documents, objects, and artifacts that collectively demonstrate their learning over a wide range of learning goals. Portfolios usually include the student's reflections and metacognitive analysis of their own learning. Portfolios are typically completed over a sustained period of time and are usually done by individual students as opposed to groups. 

Portfolios are particularly useful for evaluating how students' learning, attitudes, beliefs, and creativity grow over the span of the course. The reflective component of portfolios can be a rich form of self-feedback for students. Generally, portfolios tend to be more holistic and are often now done using ePortfolios .

Integrations

What's new?

Prototype Testing

Live Website Testing

Feedback Surveys

Interview Studies

Card Sorting

Tree Testing

In-Product Prompts

Participant Management

Automated Reports

Templates Gallery

Choose from our library of pre-built mazes to copy, customize, and share with your own users

Browse all templates

Financial Services

Tech & Software

Product Designers

Product Managers

User Researchers

By use case

Concept & Idea Validation

Wireframe & Usability Test

Content & Copy Testing

Feedback & Satisfaction

Content Hub

Educational resources for product, research and design teams

Explore all resources

Question Bank

Research Maturity Model

Guides & Reports

Help Center

Future of User Research Report

The Optimal Path Podcast

Maze Guides | Resources Hub

What is UX Research: The Ultimate Guide for UX Researchers

0% complete

Evaluative research: Key methods, types, and examples

In the last chapter, we learned what generative research means and how it prepares you to build an informed solution for users. Now, let’s look at evaluative research for design and user experience (UX).

What is evaluative research?

Evaluative research is a research method used to evaluate a product or concept and collect data to help improve your solution. It offers many benefits, including identifying whether a product works as intended and uncovering areas for improvement.

Also known as evaluation research or program evaluation, this kind of research is typically introduced in the early phases of the design process to test existing or new solutions. It continues to be employed in an iterative way until the product becomes ‘final’. “With evaluation research, we’re making sure the value is there so that effort and resources aren’t wasted,” explains Nannearl LeKesia Brown , Product Researcher at Figma.

According to Mithila Fox , Senior UX Researcher at Stack Overflow, the evaluation research process includes various activities, like content testing , assessing accessibility or desirability. During UX research , evaluation can also be conducted on competitor products to understand what solutions work well in the current market before you start building your own.

“Even before you have your own mockups, you can start by testing competitors or similar products,” says Mithila. “There’s a lot we can learn from what is and isn't working about other products in the market.”

However, evaluation research doesn’t stop when a new product is launched. For the best user experience, solutions need to be monitored after release and improved based on customer feedback.

Turn insights into impact with Maze

Create better product experiences with evaluative research powered by actionable insights from your users.

Why is evaluative research important?

Evaluative research is crucial in UX design and research, providing insights to enhance user experiences, identify usability issues, and inform iterative design improvements. It helps you:

• Refine and improve UX: Evaluative research allows you to test a solution and collect valuable feedback to refine and improve the user experience. For example, you can A/B test the copy on your site to maximize engagement with users.
• Identify areas of improvement: Findings from evaluative research are key to assessing what works and what doesn't. You might, for instance, run usability testing to observe how users navigate your website and identify pain points or areas of confusion.
• Align your ideas with users: Research should always be a part of the design and product development process . By allowing users to evaluate your product early and often you'll know whether you're building the right solution for your audience.
• Get buy-in: The insights you get from this type of research can demonstrate the effectiveness and impact of your project. Show this information to stakeholders to get buy-in for future projects.

Evaluative vs. Generative research

The difference between generative research and evaluative research lies in their focus: generative methods investigate user needs for new solutions, while evaluative research assesses and validates existing designs for improvements.

Generative and evaluative research are both valuable decision-making tools in the arsenal of a researcher. They should be similarly employed throughout the product development process as they both help you get the evidence you need.

When creating the research plan , study the competitive landscape, target audience, needs of the people you’re building for, and any existing solutions. Depending on what you need to find out, you’ll be able to determine if you should run generative or evaluative research.

Mithila explains the benefits of using both research methodologies: “Generative research helps us deeply understand our users and learn their needs, wants, and challenges. On the other hand, evaluative research helps us test whether the solutions we've come up with address those needs, wants, and challenges.”

Use generative research to bring forth new ideas during the discovery phase. And use evaluation research to test and monitor the product before and after launch.

The two types of evaluative research

There are two types of evaluative studies you can tap into: summative and formative research. Although summative evaluations are often quantitative, they can also be part of qualitative research.

Summative evaluation research

A summative evaluation helps you understand how a design performs overall. It’s usually done at the end of the design process to evaluate its usability or detect overlooked issues. You can also use a summative evaluation to benchmark your new solution against a prior one, or that of a competitor’s, and understand if the final product needs assessment. Summative evaluation can be used for outcome-focused evaluation to assess impact and effectiveness for specific outcomes—for example, how design influences conversion.

Formative evaluation research

On the other hand, formative research is conducted early and often during the design process to test and improve a solution before arriving at the final design. Running a formative evaluation allows you to test and identify issues in the solutions as you’re creating them, and improve them based on user feedback.

TL;DR: Run formative research to test and evaluate solutions during the design process, and conduct a summative evaluation at the end to evaluate the final product.

Looking to conduct UX research? Check out our list of the top UX research tools to run an effective research study.

5 Key evaluative research methods

“Evaluation research can start as soon as you understand your user’s needs,” says Mithila. Here are five typical UX research methods to include in your evaluation research process:

User surveys can provide valuable quantitative insights into user preferences, satisfaction levels, and attitudes toward a design or product. By gathering a large amount of data efficiently, surveys can identify trends, patterns, and user demographics to make informed decisions and prioritize design improvements.

Closed card sorting

Closed card sorting helps evaluate the effectiveness and intuitiveness of an existing or proposed navigation structure. By analyzing how participants group and categorize information, researchers can identify potential issues, inconsistencies, or gaps in the design's information architecture, leading to improved navigation and findability.

Tree testing

Tree testing , also known as reverse card sorting, is a research method used to evaluate the findability and effectiveness of information architecture. Participants are given a text-based representation of the website's navigation structure (without visual design elements) and are asked to locate specific items or perform specific tasks by navigating through the tree structure. This method helps identify potential issues such as confusing labels, unclear hierarchy, or navigation paths that hinder users' ability to find information.

Usability testing

Usability testing involves observing and collecting qualitative and/or quantitative data on how users interact with a design or product. Participants are given specific tasks to perform while their interactions, feedback, and difficulties are recorded. This approach helps identify usability issues, areas of confusion, or pain points in the user experience.

A/B testing

A/B testing , also known as split testing, is an evaluative research approach that involves comparing two or more versions of a design or feature to determine which one performs better in achieving a specific objective. Users are randomly assigned to different variants, and their interactions, behavior, or conversion rates are measured and analyzed. A/B testing allows researchers to make data-driven decisions by quantitatively assessing the impact of design changes on user behavior, engagement, or conversion metrics.

This is the value of having a UX research plan before diving into the research approach itself. If we were able to answer the evaluative questions we had, in addition to figuring out if our hypotheses were valid (or not), I’d count that as a successful evaluation study. Ultimately, research is about learning in order to make more informed decisions—if we learned, we were successful.

Nannearl LeKesia Brown , Product Researcher at Figma

Evaluative research question examples

To gather valuable data and make better design decisions, you need to ask the right research questions . Here are some examples of evaluative research questions:

Usability questions

• How would you go about performing [task]?
• How was your experience completing [task]?
• How did you find navigating to [X] page?
• Based on the previous task, how would you prefer to do this action instead?

Get inspired by real-life usability test examples and discover more usability testing questions in our guide to usability testing.

Product survey questions

• How often do you use the product/feature?
• How satisfied are you with the product/feature?
• Does the product/feature help you achieve your goals?
• How easy is the product/feature to use?

Discover more examples of product survey questions in our article on product surveys .

Closed card sorting questions

• Were there any categories you were unsure about?
• Which categories were you unsure about?
• Why were you unsure about the [X] category?

Find out more in our complete card sorting guide .

Evaluation research examples

Across UX design, research, and product testing, evaluative research can take several forms. Here are some ways you can conduct evaluative research:

Comparative usability testing

This example of evaluative research involves conducting usability tests with participants to compare the performance and user satisfaction of two or more competing design variations or prototypes.

You’ll gather qualitative and quantitative data on task completion rates, errors, user preferences, and feedback to identify the most effective design option. You can then use the insights gained from comparative usability testing to inform design decisions and prioritize improvements based on user-centered feedback .

Cognitive walkthroughs

Cognitive walkthroughs assess the usability and effectiveness of a design from a user's perspective.

You’ll create evaluators to identify potential points of confusion, decision-making challenges, or errors. You can then gather insights on user expectations, mental models, and information processing to improve the clarity and intuitiveness of the design .

Diary studies

Conducting diary studies gives you insights into users' experiences and behaviors over an extended period of time.

You provide participants with diaries or digital tools to record their interactions, thoughts, frustrations, and successes related to a product or service. You can then analyze the collected data to identify usage patterns, uncover pain points, and understand the factors influencing the user experience .

In the next chapters, we'll learn more about quantitative and qualitative research, as well as the most common UX research methods . We’ll also share some practical applications of how UX researchers use these methods to conduct effective research.

Generate product insights with Maze

Make actionable decisions powered by user feedback, evaluation, and research.

In the next chapters, we'll learn more about quantitative and qualitative research, as well as the most common research approaches, and share some practical applications of how UX researchers use them to conduct effective research.

Frequently asked questions

Evaluative research, also known as evaluation research or program evaluation, is a type of research you can use to evaluate a product or concept and collect data that helps improve your solution.

Quantitative vs. qualitative UX research: An overview of UX research methods

Evaluative Research Design Examples, Methods, And Questions For Product Managers

Looking for excellent evaluative research design examples?

If so, you’re in the right place!

In this article, we explore various evaluative research methods and best data collection techniques for SaaS product leaders that will help you set up your own research projects.

Sound like it’s worth a read? Let’s get right to it then!

• Evaluative research gauges how well the product meets its goals at all stages of the product development process.
• The purpose of generative research is to gain a better understanding of user needs and define problems to solve, while evaluative research assesses how successful your current product or feature is.
• Evaluation research helps teams validate ideas and estimate how good the product or feature will be at satisfying user needs, which greatly increases the chances of product success .
• Formative evaluation research sets the baseline for other kinds of evaluative research and assesses user needs.
• Summative evaluation research checks how successful the outputs of the process are against its targets.
• Outcome evaluation research evaluates if the product has had the desired effect on users’ lives.
• Quantitative research collects and analyzes numerical data like satisfaction scores or conversion rates to establish trends and interdependencies.
• Qualitative methods use non-numerical data to understand reasons for trends and user behavior.
• You can use feedback surveys to collect both quantitative and qualitative data from your target audience.
• A/B testing is a quantitative research method for choosing the best versions of a product or feature.
• Usability testing techniques like session replays or eye-tracking help PMs and designers determine how easy and intuitive the product is to use.
• Beta-testing is a popular technique that enables teams to evaluate the product or feature with real users before its launch .
• Fake door tests are a popular and cost-effective validation technique.
• With Userpilot, you can run user feedback surveys, and build user segments based on product usage data to recruit participants for interviews and beta-testing. Want to see how? Book the demo!

What is evaluative research?

Evaluative research, aka program evaluation or evaluation research, is a set of research practices aimed at assessing how well the product meets its goals .

It takes place at all stages of the product development process, both in the launch lead-up and afterward.

This kind of research is not limited to your own product. You can use it to evaluate your rivals to find ways to get a competitive edge.

Evaluative research vs generative research

Generative and evaluation research have different objectives.

Generative research is used for product and customer discovery . Its purpose is to gain a more detailed understanding of user needs , define the problem to solve, and guide product ideation .

Evaluative research, on the other hand, tests how good your current product or feature is. It assesses customer satisfaction by looking at how well the solution addresses their problems and its usability .

Why is conducting evaluation research important for product managers?

Ongoing evaluation research is essential for product success .

It allows PMs to identify ways to improve the product and the overall user experience. It helps you validate your ideas and determine how likely your product is to satisfy the needs of the target consumers.

Types of evaluation research methods

There are a number of evaluation methods that you can leverage to assess your product. The type of research method you choose will depend on the stage in the development process and what exactly you’re trying to find out.

Formative evaluation research

Formative evaluation research happens at the beginning of the evaluation process and sets the baseline for subsequent studies.

In short, its objective is to assess the needs of target users and the market before you start working on any specific solutions.

Summative evaluation research

Summative evaluation research focuses on how successful the outcomes are.

This kind of research happens as soon as the project or program is over. It assesses the value of the deliverables against the forecast results and project objectives.

Outcome evaluation research

Outcome evaluation research measures the impact of the product on the customer. In other words, it assesses if the product brings a positive change to users’ lives.

Quantitative research

Quantitative research methods use numerical data and statistical analysis. They’re great for establishing cause-effect relationships and tracking trends, for example in customer satisfaction.

In SaaS, we normally use surveys and product usage data tracking for quantitative research purposes.

Qualitative research

Qualitative research uses non-numerical data and focuses on gaining a deeper understanding of user experience and their attitude toward the product.

In other words, qualitative research is about the ‘why?’ of user satisfaction or its lack. For example, it can shed light on what makes your detractors dissatisfied with the product.

What techniques can you use for qualitative research ?

The most popular ones include interviews, case studies, and focus groups.

Best evaluative research data collection techniques

How is evaluation research conducted? SaaS PMs can use a range of techniques to collect quantitative and qualitative data to support the evaluation research process.

User feedback surveys

User feedback surveys are the cornerstone of the evaluation research methodology in SaaS.

There are plenty of tools that allow you to build and customize in-app and email surveys without any coding skills.

You use them to target specific user segments at a time that’s most suitable for what you’re testing. For example, you can trigger them contextually as soon as the users engage with the feature that you’re evaluating.

Apart from quantitative data, like the NPS or CSAT scores, it’s good practice to follow up with qualitative questions to get a deeper understanding of user sentiment towards the feature or product.

A/B testing

A/B tests are some of the most common ways of evaluating features, UI elements, and onboarding flows in SaaS. That’s because they’re fairly simple to design and administer.

Let’s imagine you’re working on a new landing page layout to boost demo bookings.

First, you modify one UI element at a time, like the position of the CTA button. Next, you launch the new version and direct half of your user traffic to it, while the remaining 50% of users still use the old version.

As your users engage with both versions, you track the conversion rate. You repeat the process with the other versions to eventually choose the best one.

Usability testing

Usability testing helps you evaluate how easy it is for users to complete their tasks in the product.

There is a range of techniques that you can leverage for usability testing :

• Guerilla testing is the easiest to set up. Just head over to a public place like a coffee shop or a mall where your target users hang out. Take your prototype with you and ask random users for their feedback.
• In the 5-second test, you allow the user to engage with a feature for 5 seconds and interview them about their impressions.
• First-click testing helps you assess how intuitive the product is and how easy it is for the user to find and follow the happy path.
• In session replays you record and analyze what the users do in the app or on the website.
• Eye-tracking uses webcams to record where users look on a webpage or dashboard and presents it in a heatmap for ease of analysis.

As with all the qualitative and quantitative methods, it’s essential to select a representative user sample for your usability testing. Relying exclusively on the early adopters or power users can skew the outcomes.

Beta testing

Beta testing is another popular evaluation research technique. And there’s a good reason for that.

By testing the product or feature prior to the launch with real users, you can gather user feedback and validate your product-market fit.

Most importantly, you can identify and fix bugs that could otherwise damage your reputation and the trust of the wider user population. And if you get it right, your beta testers can spread the word about your product and build up the hype around the launch.

How do you recruit beta testers ?

If you’re looking at expanding into new markets, you may opt for users who have no experience with your product. You can find them on sites like Ubertesters, in beta testing communities, or through paid advertising.

Otherwise, your active users are the best bet because they are familiar with the product and they are normally keen to help. You can reach out to them by email or in-app messages .

Fake door testing

Fake door testing is a sneaky way of evaluating your ideas.

Why sneaky? Well, because it kind of involves cheating.

If you want to test if there’s demand for a feature or product, you can add it to your UI or create a landing page before you even start working on it.

Next, paid adverts or in-app messages like the tooltip below, to drive traffic and engagement.

By tracking engagement with the feature, it’s easy to determine if there’s enough interest in the functionality to justify the resources you would need to spend on its development.

Of course, that’s not the end. If you don’t want to face customer rage and fury, you must always explain why you’ve stooped down to such a mischievous deed.

A modal will do the job nicely. Tell them the feature isn’t ready yet but you’re working on it. Try to placate your users by offering them early access to the feature before everybody else.

In this way, you kill two birds with one stone. You evaluate the interest and build a list of possible beta testers .

Evaluation research questions

The success of your evaluation research very much depends on asking the right questions.

Usability evaluation questions

• How was your experience completing this task?
• What technical difficulties did you experience while completing the task?
• How intuitive was the navigation?
• How would you prefer to do this action instead?
• Were there any unnecessary features?
• How easy was the task to complete?
• Were there any features missing?

Product survey research questions

• Would you recommend the product to your colleagues/friends?
• How disappointed would you be if you could no longer use the feature/product?
• How satisfied are you with the product/feature?
• What is the one thing you wish the product/feature could do that it doesn’t already?
• What would make you cancel your subscription?

How Userpilot can help product managers conduct evaluation research

Userpilot is a digital adoption platform . It consists of three main components: engagement, product analytics, and user sentiment layers. While all of them can help you evaluate your product performance, it’s the latter two that are particularly relevant.

Let’s start with the user sentiment. With Userpilot you can create customized in-app surveys that will blend seamlessly into your product UI.

You can trigger these for all your users or target particular segments.

Where do the segments come from? You can create them based on a wide range of criteria. Apart from demographics or JTBDs, you can use product usage data or survey results. In addition to the quantitative scores, you can also use qualitative NPS responses for this.

Segmentation is also great for finding your beta testers and interview participants. If your users engage with your product regularly and give you high scores in customer satisfaction surveys , they may be happy to spare some of their time to help you.

Evaluative research enables product managers to assess how well the product meets user and organizational needs, and how easy it is to use. When carried out regularly during the product development process, it allows them to validate ideas and iterate on them in an informed way.

If you’d like to see how Userpilot can help your business collect evaluative data, book the demo!

Leave a comment Cancel reply

Save my name, email, and website in this browser for the next time I comment.

Get The Insights!

The fastest way to learn about Product Growth,Management & Trends.

The coolest way to learn about Product Growth, Management & Trends. Delivered fresh to your inbox, weekly.

The fastest way to learn about Product Growth, Management & Trends.

You might also be interested in ...

8 best no-code saas tools in 2024.

Aazar Ali Shad

Top 12 Product Adoption Software in 2024

5 best no-code tools for product managers.

Your browser is unsupported

We recommend using the latest version of IE11, Edge, Chrome, Firefox or Safari.

Center for the Advancement of Teaching Excellence

Summative assessments.

Nicole Messier, CATE Instructional Designer February 7th, 2022

WHAT? Heading link Copy link

Summative assessments are used to measure learning when instruction is over and thus may occur at the end of a learning unit, module, or the entire course.

Summative assessments are usually graded, are weighted more heavily than other course assignments or comprise a substantial percentage of a students’ overall grade (and are often considered “high stakes” assessments relative to other, “lower stakes” assessments in a course), and are required assessments for the completion of a course.

Summative assessments can be viewed through two broad assessment strategies: assessments of learning and assessments as learning.

• Assessment of learning (AoL) provides data to confirm course outcomes and students the opportunity to demonstrate proficiency in the learning objectives.
• Assessment as learning (AaL) provides student ownership of learning by utilizing evidence-based learning strategies, promoting self-regulation, and providing reflective learning.

A summative assessment can be designed to provide both assessment of learning (AoL) and assessment as learning (AaL). The goal of designing for AaL and AoL is to create a summative assessment as a learning experience while ensuring that the data collected is valid and reliable.

Want to learn more about these assessment strategies? Please visit the  Resources Section – CATE website to review resources, teaching guides, and more.

Summative Assessments Heading link Copy link

Summative assessments (aol).

• Written assignments – such as papers or authentic assessments like projects or portfolios of creative work
• Mid-term exam
• Performances

Although exams are typically used to measure student knowledge and skills at the end of a learning unit, module, or an entire course, they can also be incorporated into learning opportunities for students.

Example 1 - Exam Heading link Copy link

Example 1 - exam.

An instructor decides to analyze their current multiple-choice and short-answer final exam for alignment to the learning objectives. The instructor discovers that the questions cover the content in the learning objectives; however, some questions are not at the same cognitive levels as the learning objectives . The instructor determines that they need to create some scenario questions where students are asked to analyze a situation and apply knowledge to be aligned with a particular learning objective.

The instructor also realizes that this new type of question format will be challenging for students if the exam is the only opportunity provided to students. The instructor decides to create a study guide for students on scenarios (not used in the exam) for students to practice and self-assess their learning. The instructor plans to make future changes to the quizzes and non-graded formative questions to include higher-level cognitive questions to ensure that learning objectives are being assessed as well as to support student success in the summative assessment.

This example demonstrates assessment of learning with an emphasis on improving the validity of the results, as well as assessment as learning by providing students with opportunities to self-assess and reflect on their learning.

Written assignments in any form (authentic, project, or problem-based) can also be designed to collect data and measure student learning, as well as provide opportunities for self-regulation and reflective learning. Instructors should consider using a type of grading rubric (analytic, holistic, or single point) for written assignments to ensure that the data collected is valid and reliable.

Summative Assessments (AaL) Heading link Copy link

Summative assessments (aal).

• Authentic assessments – an assessment that involves a real-world task or application of knowledge instead of a traditional paper; could involve a situation or scenario specific to a future career.
• Project-based learning – an assessment that involves student choice in designing and addressing a problem, need, or question.
• Problem-based learning – similar to project-based learning but focused on solutions to problems.
• Self-critique or peer assessment

Example 2 - Authentic Assessment Heading link Copy link

Example 2 - authentic assessment.

An instructor has traditionally used a research paper as the final summative assessment in their course. After attending a conference session on authentic assessments, the instructor decides to change this summative assessment to an authentic assessment that allows for student choice and increased interaction, feedback, and ownership.

First, the instructor introduced the summative project during the first week of class. The summative project instructions asked students to select a problem that could be addressed by one of the themes from the course. Students were provided with a list of authentic products that they could choose from, or they could request permission to submit a different product. Students were also provided with a rubric aligned to the learning objectives.

Next, the instructor created small groups (three to four students) with discussion forums for students to begin brainstorming problems, themes, and ideas for their summative project. These groups were also required to use the rubric to provide feedback to their peers at two separate time points in the course. Students were required to submit their final product, references, self-assessment using the rubric, and a reflection on the peer interaction and review.

This example demonstrates an authentic assessment as well as an assessment of learning (AoL) and assessment as learning (AaL). The validity and reliability of this summative assessment are ensured using a rubric that is focused on the learning objectives of the course and consistently utilized for the grading and feedback of the summative project. Data collected from the use of grading criteria in a rubric can be used to improve the summative project as well as the instruction and materials in the course. This summative project allows for reflective learning and provides opportunities for students to develop self-regulation skills as well as apply knowledge gained in an authentic and meaningful product.

Another way to create a summative assessment as a learning opportunity is to break it down into smaller manageable parts. These smaller parts will guide students’ understanding of expectations, provide them with opportunities to receive and apply feedback, as well as support their executive functioning and self-regulation skills.

WHY? Heading link Copy link

We know that summative assessments are vital to the curriculum planning cycle to measure student outcomes and implement continuous improvements. But how do we ensure our summative assessments are effective and equitable? Well, the answer is in the research.

Validity, Reliability, and Manageability

Critical components for the effectiveness of summative assessments are the validity, reliability, and manageability of the assessment (Khaled, 2020).

• Validity of the assessment refers to the alignment to course learning objectives. In other words, are the assessments in your course measuring the learning objectives?
• Reliability of the assessment refers to the consistency or accuracy of the assessment used. Are the assessment practices consistent from student to student and semester to semester?
• Manageability of the assessment refers to the workload for both faculty and students. For faculty, is the type of summative assessment causing a delay in timely grading and feedback to the learner? For students, is the summative assessment attainable and are the expectations realistic?

As you begin to design a summative assessment, determine how you will ensure the assessment is valid, reliable, and manageable.

Feedback & Summative Assessments

Attributes of academic feedback that improve the impact of the summative assessment on student learning (Daka, 2021; Harrison 2017) include:

• Provide feedback without or before grades.
• Once the grade is given, then explain the grading criteria and score (e.g., using a rubric to explain grading criteria and scoring).
•  Identify specific qualities in students’ work.
• Describe actionable steps on what and how to improve.
• Motivate and encourage students by providing opportunities to submit revisions or earn partial credit for submitting revised responses to incorrect answers on exams.
• Allow students to monitor, evaluate, and regulate their learning.

Additional recommendations for feedback include that feedback should be timely, frequent, constructive (what and how), and should help infuse a sense of professional identity for students (why). The alignment of learning objectives, learning activities, and summative assessments is critical to student success and will ensure that assessments are valid. And lastly, the tasks in assessments should match the cognitive levels of the course learning objectives to challenge the highest performing students while elevating lower-achieving students (Daka, 2021).

HOW? Heading link Copy link

How do you start designing summative assessments?

Summative assessments can help measure student achievement of course learning objectives as well as provide the instructor with data to make pedagogical decisions on future teaching and instruction. Summative assessments can also provide learning opportunities as students reflect and take ownership of their learning.

So how do you determine what type of summative assessment to design? And how do you ensure that summative assessment will be valid, reliable, and manageable? Let’s dive into some of the elements that might impact your design decisions, including class size, discipline, modality, and EdTech tools .

Class Size and Modality

The manageability of summative assessments can be impacted by the class size and modality of the course. Depending on the class size of the course, instructors might be able to implement more opportunities for authentic summative assessments that provide student ownership and allow for more reflective learning (students think about their learning and make connections to their experiences). Larger class sizes might require instructors to consider implementing an EdTech tool to improve the manageability of summative assessments.

The course modality can also influence the design decisions of summative assessments. Courses with synchronous class sessions can require students to take summative assessments simultaneously through an in-person paper exam or an online exam using an EdTech tool, like Gradescope or Blackboard Tests, Pools, and Surveys . Courses can also create opportunities for students to share their authentic assessments asynchronously using an EdTech tool like VoiceThread .

Major Coursework

When designing a summative assessment as a learning opportunity for major coursework, instructors should reflect on the learning objectives to be assessed and the possible real-world application of the learning objectives. In replacement of multiple-choice or short answer questions that focus on content memorization, instructors might consider creating scenarios or situational questions that provide students with opportunities to analyze and apply knowledge gained. In major coursework, instructors should consider authentic assessments that allow for student choice, transfer of knowledge, and the development of professional skills in place of a traditional paper or essay.

Undergraduate General Education Coursework

In undergraduate general education coursework, instructors should consider the use of authentic assessments to make connections to students’ experiences, goals, and future careers. Simple adjustments to assignment instructions to allow for student choice can help increase student engagement and motivation. Designing authentic summative assessments can help connect students to the real-world application of the content and create buy-in on the importance of the summative assessment.

Summative Assessment Tools

EdTech tools can help to reduce faculty workload by providing a delivery system for students to submit work as well as tools to support academic integrity.

Below are EdTech tools that are available to UIC faculty to create and/or grade summative assessments as and of learning.

Assessment Creation and Grading Tools Heading link Copy link

Assessment creation and grading tools.

• Blackboard assignments drop box and rubrics
• Blackboard quizzes and exams

Assessment creation and grading tools can help support instructors in designing valid and reliable summative assessments. Gradescope can be utilized as a grading tool for in-person paper and pencil midterm and final exams, as well as a tool to create digital summative assessments. Instructors can use AI to improve the manageability of summative assessments as well as the reliability through the use of rubrics for grading with Gradescope.

In the Blackboard learning management system, instructors can create pools of questions for both formative and summative assessments as well as create authentic assessment drop boxes and rubrics aligned to learning objectives for valid and reliable data collection.

Academic Integrity Tools

• SafeAssign (undergraduate)
•   iThenticate (graduate)
• Respondus LockDown Browser and Monitoring

Academic integrity tools can help ensure that students are meeting academic expectations concerning research through the use of SafeAssign and iThenticate as well as academic integrity during online tests and exams using Respondus Lockdown Browser and Monitoring.

Want to learn more about these summative assessment tools? Visit the EdTech section on the CATE website to learn more.

Exam Guidance

Additional guidance on online exams is available in Section III: Best Practices for Online (Remote Proctored, Synchronous) Exams in the Guidelines for Assessment in Online Environments Report , which outlines steps for equitable exam design, accessible exam technology, and effective communication for student success. The framing questions in the report are designed to guide instructors with suggestions, examples, and best practices (Academic Planning Task Force, 2020), which include:

• “What steps should be taken to ensure that all students have the necessary hardware, software, and internet capabilities to complete a remote, proctored exam?
• What practices should be implemented to make remote proctored exams accessible to all students, and in particular, for students with disabilities?
• How can creating an ethos of academic integrity be leveraged to curb cheating in remote proctored exams?
• What are exam design strategies to minimize cheating in an online environment?
• What tools can help to disincentive cheating during a remote proctored exam?
• How might feedback and grading strategies be adjusted to deter academic misconduct on exams?”

GETTING STARTED Heading link Copy link

Getting started.

The following steps will support you as you examine current summative assessment practices through the lens of assessment of learning (AoL) and assessment as learning (AaL) and develop new or adapt existing summative assessments.

• The first step is to utilize backward design principles by aligning the summative assessments to the learning objectives.
• To collect valid and reliable data to confirm student outcomes (AoL).
• To promote self-regulation and reflective learning by students (AaL).
• Format: exam, written assignment, portfolio, performance, project, etc.
• Delivery: paper and pencil, Blackboard, EdTech tool, etc.
• Feedback: general (how to improve performance), personalized (student-specific), etc.
• Scoring: automatically graded by Blackboard and/or EdTech tool or manual through the use of a rubric in Blackboard.
• The fourth step is to review data collected from summative assessment(s) and reflect on the implementation of the summative assessment(s) through the lens of validity, reliability, and manageability to inform continuous improvements for equitable student outcomes.

CITING THIS GUIDE Heading link Copy link

Citing this guide.

Messier, N. (2022). “Summative assessments.” Center for the Advancement of Teaching Excellence at the University of Illinois Chicago. Retrieved [today’s date] from https://teaching.uic.edu/resources/teaching-guides/assessment-grading-practices/summative-assessments/

ADDITIONAL RESOURCES Heading link Copy link

Academic Planning Task Force. (2020). Guidelines for Assessment in Online Learning Environments .

McLaughlin, L., Ricevuto, J. (2021). Assessments in a Virtual Environment: You Won’t Need that Lockdown Browser! Faculty Focus.

Moore, E. (2020). Assessments by Design: Rethinking Assessment for Learner Variability. Faculty Focus.

Websites and Journals

Association for the Assessment of Learning in Higher Education website 

Assessment & Evaluation in Higher Education. Taylor & Francis Online Journals

Journal of Assessment in Higher Education

REFERENCES Heading link Copy link

Daka, H., & Mulenga-Hagane, M., Mukalula-Kalumbi, M., Lisulo, S. (2021). Making summative assessment effective. 5. 224 – 237.

Earl, L.M., Katz, S. (2006). Rethinking classroom assessment with purpose in mind — Assessment for learning, assessment as learning, assessment of learning. Winnipeg, Manitoba: Crown in Right of Manitoba.

Galletly, R., Carciofo, R. (2020). Using an online discussion forum in a summative coursework assignment. Journal of Educators Online . Volume 17, Issue 2.

Harrison, C., Könings, K., Schuwirth, L. & Wass, V., Van der Vleuten, C. (2017). Changing the culture of assessment: the dominance of the summative assessment paradigm. BMC Medical Education. 17. 10.1186/s12909-017-0912-5.

Khaled, S., El Khatib, S. (2020). Summative assessment in higher education: Feedback for better learning outcomes

Research Design in Evaluation: Choices and Issues

Cite this chapter.

• Keith Tones ,
• Sylvia Tilford &
• Yvonne Keeley Robinson  

179 Accesses

1 Citations

One of the purposes of this book is to stimulate reflection on the meaning of effectiveness and efficiency in health education. Moreover, although it is not concerned to offer detailed guidance on the measurement of success, it may well be worthwhile reminding readers of what is involved in evaluating programmes. Accordingly this chapter will provide a condensed account of the most common research designs used in evaluation . In so doing it will seek to underline some of the important issues involved in selecting an evaluation strategy and perhaps help potential evaluators make a pragmatic choice of method. It should also serve as a reminder that evaluation is not a mere technical activity: it incorporates philosophical issues and requires understanding of the purpose of research and the meaning of success. Above all, the reader should recall that there are often not inconsiderable difficulties involved in gaining unequivocal answers to the apparently simple question ‘Has this health education programme been effective?’

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as EPUB and PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Unable to display preview.  Download preview PDF.

Jamieson, I. (1984) Evaluation: a case of research in chains? in: The Politics and Ethics of Evaluation , (ed. C. Adelman), Croom Helm, London.

Google Scholar  

Campbell, D. T. (1969) Reforms as experiments. American Psychologist , 24 , 409–29.

Article   Google Scholar  

Fisher, R. A. (1935) The Design of Experiments , Oliver and Boyd, London.

Shaddock Evans, E. G. (1962) The design of teaching experiments in education, Educational Research , 5 , 37–52.

Roethlisberger, F. J. and Dickson, W. J. (1939) Management and the Worker , Harvard University Press, Cambridge.

Campbell, D. T. and Stanley, J.C. (1963) Experimental and Quasi-Experimental Designs for Research , Rand University, Chicago.

Cronbach, L. (1983) Designing Evaluation of Educational and Social Programmes , Jossey Bass, San Francisco.

Marsh, C. (1982) The Survey Method: The Contribution of Surveys to Sociological Explanation , George Allen and Unwin, London.

Nelson, S. C, Budd, R. J. and Eiser, J. R. (1985) The Avon prevalence study: a survey of cigarette smoking in secondary school children, Health Education Journal , 44 , 12–14.

Feuerstein, M. T. (1986) Partners in Evaluation. Evaluating Development and Community Programmes with Participants . Macmillan, London.

Parlett, M. and Hamilton, D. (1972) Evaluation as Illumination , Centre for Research in Educational Sciences, University of Edinburgh.

Hammersley, M. and Atkinson, P. (1983) Ethnography Principles in Practice , Tavistock, London.

Glaser, B. and Strauss, A. (1967) The Discovery of Grounded Theory , Aldine, Chicago.

Malinowski, B. (1922) Argonauts of the Western Pacific , Routledge and Kegan Paul, London.

Scriven, M. (1973) Goal-free evaluation, in: School Evaluation: The Politics and Process (ed. E. R. House), McCutchan, California.

Chalmers, A. F. (1982) What is This Thing Called Science? (2nd edn) Open University Press, Milton Keynes.

Kuhn, T. S. (1970) The Structure of Scientific Revolutions , University of Chicago Press, Chicago.

McQueen, D. V. (1986) Health education research: the problem of linkages, Health Education Research , 1 , 289–94.

Wright Mills, C. (1970) Appendix: on intellectual craftsmanship, in: The Sociological Imagination , Penguin, London.

Barnes, J. A. (1979) Who Should Know What? Penguin, London.

Buhner, M. (Ed) (1982) Social Research Ethics , Macmillan, London.

Fichter, J. H. and Kolb, W. L. (1953) Ethical limitations in sociological reporting, American Sociological Review , 18 , 544–50.

American Sociological Association (1968) Towards a code of ethics for sociologists, American Sociologists , 3 , 316–8.

Reason, P. and Rowan, J. (1981) Human Inquiry. A Sourcebook of New Paradigm Research , Wiley, London.

Galliher, J. (1973) The protection of human subjects: a re-examination of the professional code of ethics, American Sociologist , 8 , 93–100.

PubMed   Google Scholar  

Becker, H. (1967) Whose side are we on? Social Problems , 14 , 239–47.

Download references

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 1990 Tones, Tilford and Robinson

About this chapter

Tones, K., Tilford, S., Robinson, Y.K. (1990). Research Design in Evaluation: Choices and Issues. In: Health Education. Springer, Boston, MA. https://doi.org/10.1007/978-1-4899-3230-3_2

Download citation

DOI : https://doi.org/10.1007/978-1-4899-3230-3_2

Publisher Name : Springer, Boston, MA

Print ISBN : 978-0-412-32980-7

Online ISBN : 978-1-4899-3230-3

eBook Packages : Springer Book Archive

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Gen Intern Med
• v.21(Suppl 2); 2006 Feb

The Role of Formative Evaluation in Implementation Research and the QUERI Experience

Cheryl b stetler.

1 Independent Consultant, Amherst, MA, USA

Marcia W Legro

2 VA Puget Sound Health Care System, Seattle, WA, USA

3 University of Washington, Seattle, WA, USA

Carolyn M Wallace

Candice bowman.

4 VA San Diego Healthcare System, San Diego, CA, USA

Marylou Guihan

5 Edward Hines, Jr. VA Healthcare System, Hines, IL, USA

Hildi Hagedorn

6 Minneapolis VA Medical Center, Minneapolis, MN, USA

Barbara Kimmel

7 Baylor College of Medicine, Houston, TX, USA

8 Houston Veterans Affairs Medical Center, Houstan, TX, USA

Nancy D Sharp

Jeffrey l smith.

9 Central Arkansas Veterans Healthcare System, Little Rock, AR, USA

This article describes the importance and role of 4 stages of formative evaluation in our growing understanding of how to implement research findings into practice in order to improve the quality of clinical care. It reviews limitations of traditional approaches to implementation research and presents a rationale for new thinking and use of new methods. Developmental, implementation-focused, progress-focused, and interpretive evaluations are then defined and illustrated with examples from Veterans Health Administration Quality Enhancement Research Initiative projects. This article also provides methodologic details and highlights challenges encountered in actualizing formative evaluation within implementation research.

As health care systems struggle to provide care based on well-founded evidence, there is increasing recognition of the inherent complexity of implementing research into practice. Health care managers and decision makers find they need a better understanding of what it takes to achieve successful implementation, and they look to health care researchers to provide this information. Researchers in turn need to fill this need through collection of new, diverse sets of data to enhance understanding and management of the complex process of implementation.

A measurement approach capable of providing critical information about implementation is formative evaluation (FE). Formative evaluation, used in other social sciences, is herein defined as a rigorous assessment process designed to identify potential and actual influences on the progress and effectiveness of implementation efforts . Formative evaluation enables researchers to explicitly study the complexity of implementation projects and suggests ways to answer questions about context, adaptations, and response to change.

The Department of Veterans Affairs (VA) Quality Enhancement Research Initiative (QUERI) has integrated FE into its implementation program. 1 – 3 This article introduces QUERI and its implementation focus. It then describes research challenges that call for the use of FE in this specialized field of study, reviews FE relative to QUERI implementation research, identifies 4 evaluative stages, and presents challenges to the conduct of FE.

THE VETERAN HEALTH ADMINISTRATION'S QUERI PROGRAM

The Quality Enhancement Research Initiative, begun in 1998, is a comprehensive, data-driven, outcomes-based, and output-oriented improvement initiative. 2 , 3 It focuses on identification and implementation of empirically based practices for high-risk/high-volume conditions among the veteran population and on the evaluation and refinement of these implementation efforts. 3 The Quality Enhancement Research Initiative's innovative approach 1 – 4 calls upon researchers to work toward rapid, significant improvements through the systematic application of best clinical practices. It also calls upon researchers to study the implementation process to enhance and continuously refine these quality improvement (QI) efforts. 1 – 4

Classic intervention research methods 5 , 6 provide the means to evaluate targeted outcomes of implementation/QI efforts. From an evaluation perspective, studies using intervention designs, such as a cluster-randomized trial or quasi-experimental approaches, routinely include a summative evaluation . Summative evaluation is a systematic process of collecting data on the impacts, outputs, products, or outcomes hypothesized in a study. 7 Resulting data provide information on the degree of success, effectiveness, or goal achievement of an implementation program.

In an action-oriented improvement program, such as QUERI, summative data are essential but insufficient to meet the needs of implementation/QI researchers. Evaluative information is needed beyond clinical impact of the change effort and beyond discovering whether a chosen adoption strategy worked. Implementation researchers need to answer critical questions about the feasibility of implementation strategies, degree of real-time implementation, status and potential influence of contextual factors, response of project participants, and any adaptations necessary to achieve optimal change. Formative evaluation provides techniques for obtaining such information and for overcoming limitations identified in early implementation/QI studies.

NEED FOR FE IN IMPLEMENTATION/QI RESEARCH

The RE-AIM framework of Glasgow and colleagues highlights critical information that is missing from current research publications—i.e.,information needed to evaluate a study's potential for translation and public health impact . 8 , 9 Such information includes the efficacy/effectiveness of an intervention, its reach relative to actual/representative subject participation rate, its adoption relative to actual/representative setting participation rate, its implementation or intervention fidelity , and its maintenance over time.

The focus of the RE-AIM framework is the study of health promotion interventions. Similar issues must be addressed during implementation research if potential adopters are to replicate critical implementation processes. In addition, implementation researchers need to capture in-depth information on participant and contextual factors that facilitate or hinder successful implementation. Such factors can be used during the project to optimize implementation and inform post hoc interpretation.

As implementation efforts can be a relatively messy and complex process, traditional study designs alone are often inadequate to the task of obtaining evaluative information. For example, randomized clinical trials (RCT) may leave questions important to system-wide uptake of targeted research unanswered. As Stead et al. 10 , 11 suggest, traditional intervention research can fail to “capture the detail and complexity of intervention inputs and tactics” (10, p. 354), thereby missing the true nature of interventions as well as significant organizational factors important for replication. 10 , 11

Another argument for performing FE has been highlighted in guideline/QI literature, i.e., the need to address potential interpretive weaknesses. Such weaknesses relate to a failure to account for key elements of the implementation process and may lead to unexplainable and/or poor results. For example, Ovretveit and Gustafson 12 identified implementation assessment failure, explanation failure , and outcome attribution failure . Implementation assessment failure can lead to a “Type III” error, where erroneous study interpretations occur because the intervention was not implemented as planned. 12 , 13 Explanation and outcome attribution relate to failures to explore the black box of implementation. Specifically, what actually did/did not happen within the study relative to the implementation plan, and what factors in the implementation setting, anticipated or unanticipated, influenced the actual degree of implementation? By failing to collect such data, potential study users have little understanding of a particular implementation strategy. For example, 1 study regarding opinion leadership did not report the concurrent implementation of standing orders. 14

Use of a traditional intervention design does not obviate collection of the critical information cited above. Rather, complementary use of FE within an experimental study can create a dual or hybrid style approach for implementation research. 15 The experimental design is thus combined with descriptive or observational research that employs a mix of qualitative and quantitative techniques, creating a richer dataset for interpreting study results.

FORMATIVE EVALUATION WITHIN QUERI

As with many methodologic concepts, there is no single definition/approach to FE. In fact, as Dehar et al. 16 stated, there is a decided “lack of clarity and some disagreement among evaluation authors as to the meaning and scope” of related concepts (16, p. 204; see Table 1 for a sampling). Variations include differences in terminology, e.g., an author may refer to FE , process evaluation , or formative research . 16 , 17

A Spectrum of Definitions of Formative Evaluation

Given a mission to make rapid, evidence-based improvements to achieve better health outcomes, the authors have defined FE as a rigorous assessment process designed to identify potential and actual influences on the progress and effectiveness of implementation efforts . Related data collection occurs before, during, and after implementation to optimize the potential for success and to better understand the nature of the initiative, need for refinements, and the worth of extending the project to other settings. This approach to FE incorporates aspects of the last 2 definitions in Table 1 and concurs with the view that formative connotes action. 16 In QUERI, this action focus differentiates FE from “process” evaluations where data are not intended for concurrent use.

Various uses of FE for implementation research are listed in Table 2 . Uses span the timeframe or stages of a project, i.e., development/diagnosis, implementation, progress, and interpretation. Within QUERI, these stages are progressive, integrated components of a single hybrid project. Each stage is described below, in the context of a single project, and illustrated by QUERI examples ( Tables 3 , ​ ,4, 4 , and ​ and6 6 – 10 ). Each table provides an example of 1 or more FE stages. However, as indicated in some of the examples, various evaluative activities can serve multiple stages, which then merge in practice. Formative evaluation at any stage requires distinct plans for adequate measurement and analysis.

Potential Uses of Formative Evaluation 10 , 13 , 16 , 20 – 27

An Example of Developmental FE

QUERI, quality enhancement research initiative; SCI, spinal cord injury; VHA, Veterans Health Administration; CCR,computerized clinical reminder; FE, formative evaluation.

Implementation-Focused FE

QUERI, quality enhancement research initiative; FE, formative evaluation; CHF, congestive heart failure.

Implementation and Progress-Focused FE

QUERI, quality enhancement research initiative; FE, formative evaluation; QI, quality improvement.

An Illustrative, Potential FE

QUERI, quality enhancement research initiative; FE, formative evaluation.

Developmental Evaluation

Developmental evaluation occurs during the first stage of a project and is termed a diagnostic analysis . 1 , 28 It is focused on enhancing the likelihood of success in the particular setting/s of a project, and involves collection of data on 4 potential influences: (a) actual degree of less-than-best practice; (b) determinants of current practice; (c) potential barriers and facilitators to practice change and to implementation of the adoption strategy; and (d) strategy feasibility, including perceived utility of the project. (Note: studies conducted to obtain generic diagnostic information prior to development of an implementation study are considered formative research, not FE. Even if available, a diagnostic analysis is suggested given the likelihood that generically identified factors will vary across implementation sites.)

Activity at this stage may involve assessment of known prerequisites or other factors related to the targeted uptake of evidence, e.g., perceptions regarding the evidence, attributes of the proposed innovation, and/or administrative commitment. 11 , 21 , 29 – 31 Examples of formative diagnostic tools used within QUERI projects include organizational readiness and attitude/belief surveys 32 , 33 (also see Tables 3 and ​ and7). 7 ). Such developmental data enable researchers to understand potential problems and, where possible, overcome them prior to initiation of interventions in study sites.

Developmental/Implementation/Progress FE

QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.

In addition to information available from existent databases about current practice or setting characteristics, formative data can be collected from experts and representative clinicians/administrators. For example, negative unintended consequences might be prospectively identified by key informant or focus group interviews. This participatory approach may also facilitate commitment among targeted users. 34

Implementation-Focused Evaluation

This type of FE occurs throughout implementation of the project plan. It focuses on analysis of discrepancies between the plan and its operationalization and identifies influences that may not have been anticipated through developmental activity. As Hulscher et al. note in a relevant overview of “process” evaluation, FE allows “researchers and implementers to (a) describe the intervention in detail, (b) evaluate and measure actual exposure to the intervention, and (c) describe the experience of those exposed (13, p. 40)”— concurrently. It also focuses on the dynamic context within which change is taking place, an increasingly recognized element of implementation. 37 – 40

Implementation-focused formative data enable researchers to describe and understand more fully the major barriers to goal achievement and what it actually takes to achieve change, including the timing of project activities. By describing the actuality of implementation, new interventions may be revealed. In terms of timing, formative data can clarify the true length of time needed to complete an intervention, as failure to achieve results could relate to insufficient intervention time.

Implementation-focused formative data also are used to keep the strategies on track and as a result optimize the likelihood of affecting change by resolving actionable barriers, enhancing identified levers of change, and refining components of the implementation interventions. Rather than identify such modifiable components on a post hoc basis, FE provides timely feedback to lessen the likelihood of type III errors (see Tables 4 , ​ ,6, 6 , ​ ,7, 7 , and ​ and9 9 ).

Implementation/Interpretive FE

QUERI, quality enhancement research initiative; FE, formative evaluation; CR, clinical reminder; SAS, site activation scale.

In summary, FE data collected at this stage offer several advantages. They can (a) highlight actual versus planned interventions, (b) enable implementation through identification of modifiable barriers, (c) facilitate any needed refinements in the original implementation intervention, (d) enhance interpretation of project results, and (e) identify critical details and guidance necessary for replication of results in other clinical settings.

Measurement within this stage can be a simple or complex task. Table 5 describes several critical issues that researchers should consider. As with other aspects of FE, both quantitative and qualitative approaches can be used.

Critical Measures of Implementation

Progress-Focused Evaluation

This type of FE occurs during implementation of study strategies, but focuses on monitoring impacts and indicators of progress toward goals. The proactive nature of FE is emphasized, as progress data become feedback about the degree of movement toward desired outcomes. Using implementation data on dose, intensity, and barriers, factors blocking progress may be identified. Steps can then be taken to optimize the intervention and/or reinforce progress via positive feedback to key players. As Krumholz and Herrin 49 suggest, waiting until implementation is completed to assess results “obscures potentially important information … about trends in practice during the study [that] could demonstrate if an effort is gaining momentum—or that it is not sustainable” (see Tables 6 and ​ and7 7 ).

Interpretive Evaluation

This stage is usually not considered a type of FE but deserves separate attention, given its role in the illumination of the black box of implementation/change. Specifically, FE data provide alternative explanations for results, help to clarify the meaning of success in implementation, and enhance understanding of an implementation strategy's impact or “worth.” Such “black box” interpretation occurs through the end point triangulation of qualitative and quantitative FE data, including associational relationships with impacts.

Interpretive FE uses the results of all other FE stages. In addition, interpretive information can be collected at the end of the project about key stakeholder experiences. Stakeholders include individuals expected to put evidence into practice as well as those individuals expected to support that effort. These individuals can be asked about their perceptions of the implementation program, its interventions, and changes required of them and their colleagues. 10 , 13 , 27 , 38 , 46 , 54 Information can be obtained on stakeholder views regarding (a) usefulness or value of each intervention, (b) satisfaction or dissatisfaction with various aspects of the process, (c) reasons for their own program-related action or inaction, (d) additional barriers and facilitators, and (e) recommendations for further refinements.

Information can also be obtained regarding the degree to which stakeholders believe the implementation project was successful, as well as the overall “worth” of the implementation effort. Statistical significance will be calculated using the summative data. However, as inferential statistical significance does not necessarily equate with clinical significance, it is useful to obtain perceptions of stakeholders relative to the “meaning” of statistical findings. For some stakeholders, this meaning will be placed in the context of the cost of obtaining the change relative to its perceived benefits (see Tables 8 – 10 ).

Interpretive FE

QUERI, quality enhancement research initiative; FE, formative evaluation; QI, quality improvement; NH, mental health; ATIP, antipsychotic treatment improvement program.

Formative evaluation, as a descriptive assessment activity, does not per se test hypotheses. However, within an experimental study, in-depth data from a concurrent FE can provide working hypotheses to explain successes or failures, particularly when the implementation and evaluation plans are grounded in a conceptual framework. 55 – 57 In this respect, interpretive FE may be considered as case study data that contribute to theory building. 58 Overall, FE data may provide evidence regarding targeted components of a conceptual framework, insights into the determinants of behavior or system change, and hypotheses for future testing.

CHALLENGES OF CONDUCTING FE

Formative evaluation is a new concept as applied to health services research and as such presents multiple challenges. Some researchers may need help in understanding how FE can be incorporated into a study design. Formative evaluation is also a time-consuming activity and project leaders may need to be convinced of its utility before committing study resources. In addition, much is yet to be learned about effective approaches to the following types of issues:

• In the well-controlled RCT, researchers do not typically modify an experimental intervention once approved. However, in improvement-oriented research, critical problems that prevent an optimal test of the planned implementation can be identified and resolved. Such actions may result in alterations to the original plan. The challenge for the researcher is to identify that point at which modifications create a different intervention or add an additional intervention. Likewise, when the researcher builds in “local adaptation,” the challenge is to determine its limits or clarify statistical methods available to control for the differences. An implementation framework and clear identification of the underlying conceptual nature of each intervention can facilitate this process. As Hawe et al. 43 suggest, the researcher has to think carefully about the “essence of the intervention” in order to understand the actual nature of implementation and the significance of formative modifications.
• Implementation and QI researchers may encounter the erroneous view that FE involves only qualitative research or that it is not rigorous, e.g., that it consists of “just talking to a few people”. However, FE does not lack rigor nor is it simply a matter of qualitative research or a specific qualitative methodology. Rather, FE involves selecting among rigorous qualitative and quantitative methods to accomplish a specific set of aims, with a plan designed to produce credible data relative to explicit formative questions. 61
• A critical challenge for measurement planning is selection or development of methods that yield quantitative data for the following types of issues: (a) assessment of associations between outcome findings and the intensity, dose, or exposure to interventions and (b) measurement of the adaptations of a “standard” protocol across diverse implementation settings. 62 Whether flexibility is a planned or unplanned component of a study, it should be measured in some consistent, quantifiable fashion that enables cross-site comparisons. Goal attainment scaling is 1 possibility. 47 , 48
• A final issue facing implementation researchers is how to determine the degree to which FE activities influence the results of an implementation project. If FE itself is an explicit intervention, it will need to be incorporated into recommendations for others who wish to replicate the study's results. More specifically, the researcher must systematically reflect upon why formative data were collected, how they were used, by whom they were used, and to what end. For example, to what extent did FE enable refinement to the implementation intervention such that the likelihood of encountering barriers in the future is adequately diminished? Or, in examining implementation issues across study sites, to what extent did FE provide information that led to modifications at individual sites? If the data and subsequent adjustments at individual sites were deemed critical to project success, upon broader dissemination to additional sites, what specific FE activities should be replicated, and by whom?

Formative evaluation is a study approach that is often key to the success, interpretation, and replication of the results of implementation/QI projects. Formative evaluation can save time and frustration as data highlight factors that impede the ability of clinicians to implement best practices. It can also identify at an early stage whether desired outcomes are being achieved so that implementation strategies can be refined as needed; it can make the realities and black box nature of implementation more transparent to decision makers; and it can increase the likelihood of obtaining credible summative results about effectiveness and transferability of an implementation strategy. Formative evaluation helps to meet the many challenges to effective implementation and its scientific study, thereby facilitating integration of research findings into practice and improvement of patient care.

Acknowledgments

The work reported here was supported by the Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.

Study record managers: refer to the Data Element Definitions if submitting registration or results information.

Search for terms

• Advanced Search
• See Studies by Topic
• See Studies on Map
• How to Search
• How to Use Search Results
• How to Find Results of Studies
• How to Read a Study Record

• Learn About Studies
• Other Sites About Studies
• Glossary of Common Site Terms

• Submit Studies to ClinicalTrials.gov PRS
• Why Should I Register and Submit Results?
• FDAAA 801 and the Final Rule
• How to Apply for a PRS Account
• How to Register Your Study
• How to Edit Your Study Record
• How to Submit Your Results
• Frequently Asked Questions
• Support Materials
• Training Materials

• Selected Publications
• Clinical Alerts and Advisories
• Trends, Charts, and Maps
• Downloading Content for Analysis

• ClinicalTrials.gov Background
• About the Results Database
• History, Policies, and Laws
• ClinicalTrials.gov Modernization
• Media/Press Resources
• Linking to This Site
• Terms and Conditions
• Search Results
• Study Record Detail

Spire Health Remote Patient Monitor Human Factors Summative Study Protocol

• Study Details
• Tabular View
• No Results Posted

The summative testing will address user interactions with all components of the Spire Remote Patient Monitor related to tasks associated with the user group, including the Medical Health Tags, SpireHealth Mobile Application and Healthcare Provider Dashboard. This protocol specifies the required testing methods and associated acceptance criteria, as well as other information necessary to collect that evidence.

This human factors summative test is aimed to uncover the following:

• Determine if the final system design is safe and effective for use by the intended users in the intended use environment
• Investigation of any use errors related to risk observed
• To identify any potential new use errors relating to risk Background

Empirical activities in the form of a Cognitive Expert Review Panel was conducted to evaluate design iterations on critical tasks prior to summative testing. The results of these activities were used to improve UI design and inform analytical analyses such as task analysis, PCA analysis and use error analysis (DOC-75057 Use Error Analysis) This validation includes all Spire Remote Patient Monitor system interfaces associated with the primary consumer user groups.

The Spire Remote Patient Monitor that are the subject of this validation need to be supported by the following devices:

• SpireHealth Mobile Application running on a compatible smart phone (Apple iPhone or Android phone)
• Medical Health Tag device
• Spire Healthcare Provider Dashboard running in a browser on a personal computer

Prior to this summative testing there will be no training activities. Support and help are embedded within the app experience and Instructions for Use (IFU).

Seventeen (17) users will be recruited who comprise the single Spire Remote Patient Monitor user group: adults for whom remote monitoring has been indicated by a health professional. At least 15 usable data points will be collected. This sample size is based on the recommendation number of participants in IEC 62366: 2015, Part 1 and Part 2. The increased participant size is also to allow for cancellations and other test anomalies (e.g., equipment failures).

Session Description: 60 minutes, one-on-one session

Session Activities

• Introduction and background questions: 5 minutes
• Device in-use evaluation (observations and questions): 45 minutes
• The final follow-up and closing: 10 minutes

The Facilitator's Script is allows for consistent interaction between the facilitators and the participants. As per the facilitator's discretion, any deviation to the exact script will be based on real-time responses to the session flow and responding the individual interaction. The script will have a list of questions specific to each scenario designed to help the facilitator/observer refine the information, to improve consistency of reporting, and assist the facilitator/observer in root cause analysis.

Inclusion Criteria:

• Must indicate that they are interested in using a smartphone and/or wearable fitness device for tracking their health
• High school degree or equivalent (e.g., GED) or less - at least 2
• Some college but no degree or associate degree - at least 4
• Age 22-49 - at least 5
• Age 50-65 - at least 5
• Age Over 65 - at least 2
• Near even split of men and women
• Almost never limited by my physical condition
• Rarely limited by my physical condition
• Occasionally limited by my physical condition
• Often limited by my physical condition
• Almost always limited by my physical condition
• Users with iPhone experience
• Smart Phone Savviness
• No more than 7 users who own use a wearable fitness device
• Wearable Fitness Device savviness

Exclusion Criteria:

• Participant currently works in tech industry
• Participant or a member of his/her household currently works for any company that develops, manufactures, or sells smartphones or smart wearable fitness devices
• Participant has participated in market or user research related to a medical device within the last 6 months and has participated in more than 2 in the last 3 years
• Participants are trained clinicians or healthcare providers
• For Patients and Families
• For Researchers
• For Study Record Managers
• Customer Support
• Accessibility
• Viewers and Players
• Freedom of Information Act
• HHS Vulnerability Disclosure
• U.S. National Library of Medicine
• U.S. National Institutes of Health
• U.S. Department of Health and Human Services

• Open access
• Published: 14 May 2024

Evaluation of the feasibility of a midwifery educator continuous professional development (CPD) programme in Kenya and Nigeria: a mixed methods study

• Duncan N. Shikuku 1 , 2 ,
• Hauwa Mohammed 3 ,
• Lydia Mwanzia 4 ,
• Alice Norah Ladur 2 ,
• Peter Nandikove 5 ,
• Alphonce Uyara 6 ,
• Catherine Waigwe 7 ,
• Lucy Nyaga 1 ,
• Issak Bashir 8 ,
• Eunice Ndirangu 9 ,
• Carol Bedwell 2 ,
• Sarah Bar-Zeev 10 &
• Charles Ameh 2 , 11 , 12  

BMC Medical Education volume  24 , Article number:  534 ( 2024 ) Cite this article

35 Accesses

9 Altmetric

Metrics details

Midwifery education is under-invested in developing countries with limited opportunities for midwifery educators to improve/maintain their core professional competencies. To improve the quality of midwifery education and capacity for educators to update their competencies, a blended midwifery educator-specific continuous professional development (CPD) programme was designed with key stakeholders. This study evaluated the feasibility of this programme in Kenya and Nigeria.

This was a mixed methods intervention study using a concurrent nested design. 120 randomly selected midwifery educators from 81 pre-service training institutions were recruited. Educators completed four self-directed online learning (SDL) modules and three-day practical training of the blended CPD programme on teaching methods (theory and clinical skills), assessments, effective feedback and digital innovations in teaching and learning. Pre- and post-training knowledge using multiple choice questions in SDL; confidence (on a 0–4 Likert scale) and practical skills in preparing a teaching a plan and microteaching (against a checklist) were measured. Differences in knowledge, confidence and skills were analysed. Participants’ reaction to the programme (relevance and satisfaction assessed on a 0–4 Likert scale, what they liked and challenges) were collected. Key informant interviews with nursing and midwifery councils and institutions’ managers were conducted. Thematic framework analysis was conducted for qualitative data.

116 (96.7%) and 108 (90%) educators completed the SDL and practical components respectively. Mean knowledge scores in SDL modules improved from 52.4% (± 10.4) to 80.4% (± 8.1), preparing teaching plan median scores improved from 63.6% (IQR 45.5) to 81.8% (IQR 27.3), and confidence in applying selected pedagogy skills improved from 2.7 to 3.7, p  < 0.001. Participants rated the SDL and practical components of the programme high for relevance and satisfaction (median, 4 out of 4 for both). After training, 51.4% and 57.9% of the participants scored 75% or higher in preparing teaching plans and microteaching assessments. Country, training institution type or educator characteristics had no significant associations with overall competence in preparing teaching plans and microteaching ( p  > 0.05). Qualitatively, educators found the programme educative, flexible, convenient, motivating, and interactive for learning. Internet connectivity, computer technology, costs and time constraints were potential challenges to completing the programme.

The programme was feasible and effective in improving the knowledge and skills of educators for effective teaching/learning. For successful roll-out, policy framework for mandatory midwifery educator specific CPD programme is needed.

Peer Review reports

Introduction

Quality midwifery education underpins the provision of quality midwifery care and is vital for the health and well-being of women, infants, and families [ 1 ]. The recent State of the World’s Midwifery report (SoWMy) (2021) indicates that urgent investments are needed in midwifery, especially quality midwifery education, to improve health outcomes for women and neonates. Despite evidence to support midwifery, midwifery education and training is grossly underfunded in low- and middle-income countries (LMICs) with variation in the quality, content and duration of content between and within countries [ 2 ]. Barriers to achieving quality education are: inadequate content, lack of learning and teaching materials, insufficient and poorly trained educators and weak regulation, midwifery educators having no connection with clinical practice or opportunities for updating their knowledge or skills competencies [ 3 , 4 ].

The WHO, UNFPA, UNICEF and the International Confederation of Midwives’ (ICM) seven-step action plan to strengthen quality midwifery education, and ICM’s four pillars for midwives to achieve their potential emphasize strengthening midwifery faculty to teach students as a key priority [ 4 , 5 ]. Consequently, ICM recommends that (i) at least 50% of midwifery education curriculum should be practise-based with opportunities for clinical experience, (ii) midwifery faculty should use fair, valid and reliable formative and summative assessment methods to measure student performance and progress in learning and (iii) midwifery programmes have sufficient and up-to-date teaching and learning resources and technical support for virtual/distance learning to meet programme needs [ 6 ]. To achieve this, WHO’s Midwifery Educator Core Competencies and ICM’s Global Standards for Midwifery Education provide core competencies that midwifery educators must possess for effective practice [ 6 , 7 ]. The WHO’s global midwifery educator survey in 2018–2019 reported that fewer than half of the educators (46%) were trained or accredited as educators [ 5 ]. Educators are important determinants of quality graduates from midwifery programmes [ 7 ]. However, the survey identified that none of the educators felt confident in all of WHO’s midwifery educator core competencies [ 5 ]. Further evidence shows that many midwifery educators are more confident with theoretical classroom teaching than clinical teaching despite advances in teaching methods and have low confidence in facilitating online/virtual teaching and learning [ 4 , 8 , 9 ]. To remain competent, design and deliver competency-based curriculum and strengthen midwifery practice, ICM and WHO emphasize that midwifery faculty should engage in ongoing professional development as a midwifery practitioner, teacher/lecturer and leader [ 6 , 10 , 11 ]. However in many settings there is inadequate provision or access to faculty development opportunities [ 12 ].

Continuous professional development (CPD)

Continuous professional development has been defined as the means by which members of the profession maintain, improve and broaden their knowledge, expertise, and competence, and develop the personal and professional qualities required throughout their professional lives [ 13 ]. This can be achieved through multiple formal educational pathways based on the ICM Global Standards for Midwifery Education whilst incorporating the ICM Essential Competencies for Basic Midwifery Practice [ 6 , 14 ]. There are formal CPD activities where there is structured learning that often follows set curricula, usually approved by independent accreditation services or informal CPD that is usually self-directed learning. Participating in accredited CPD programmes is beneficial to the profession. A requirement of regular CPD renewal by a country to maintain licensure ensures an up-to-date, relevant nursing and midwifery workforce [ 15 ] and increases the legitimacy of CPD [ 16 ]. Structured learning (direct or distant), mandatory training, attending workshops and conferences, accredited college/university courses and trainings, research and peer review activities are opportunities for CPD [ 17 ]. Importantly, these CPD programmes are essential for safe, competent and effective practice that is essential to the universal health coverage (UHC) & maternal and newborn health SDGs agenda particularly in developing countries [ 18 , 19 ].

Whilst regulatory bodies and employers in many countries have requirements for midwives to complete CPD programmes and activities, these programmes and supporting activities are found to be ineffective if CPD is irrelevant to the practitioners’ practice setting, attended only because of monetary or non-monetary benefits, geared towards improving a skill for which there is no demonstrated need, and taken only to meet regulatory requirements rather than to close a competency gap [ 20 ]. In most LMICs, midwifery licensure is permanent, without obligation to demonstrate ongoing education or competence [ 15 ]. Consequently, CPD processes are not in place, and if in place, not fully utilised. A systematic review on CPD status in WHO regional office for Africa member states reported that nurses and midwives are required to attend formalised programmes delivered face-to-face or online, but only16 out of 46 (34.7%) member states had mandatory CPD programmes [ 15 ]. This underscores the need for designing regulator approved midwifery educator CPD programmes to improve the quality of midwifery education in LMICs.

Modes and approaches for delivery of CPD

Face-to-face contact is a common mode of delivery of CPD although mHealth is an emerging platform that increases access, particularly to nurses and midwives in rural areas [ 12 , 21 ]. Emerging platforms and organisations such as World Continuing Education Alliance (WCEA) offer mHealth learning opportunities in LMICs for skilled health personnel to access CPD resources that can improve health care provider knowledge and skills and potentially positively impact healthcare outcomes [ 22 ]. Although there is evidence of capacity building initiatives and CPD for midwifery educators in LMICs [ 23 ], these have been largely delivered as part of long duration (2-year) fellowship programmes and led by international organisations. In addition, these programmes have largely focused on curriculum design, leadership, management, research, project management and programme evaluation skills in health professions education with little on teaching and learning approaches and assessment for educators [ 24 , 25 , 26 ]. Successful CPD initiatives should be (i) accredited by the national regulatory bodies (Nursing and Midwifery Councils); (ii) multifaceted and provide different types of formal and informal learning opportunities and support; (iii) combine theory and clinical practice to develop the knowledge, skills and attitudes and (iv) must be adapted to fit the local context in which participants work and teach to ensure local ownership and sustainability of the initiatives [ 16 ].

Short competency-based blended trainings for educators improve their competence and confidence in delivering the quality midwifery teaching. However, systems for regular updates to sustain the competencies are lacking [ 27 , 28 ]. Evidence on effectiveness of the available CPD initiatives is limited. Even where these initiatives have been evaluated, this has largely focused on the outcomes of the programmes and little attention on the feasibility and sustainability of such programmes in low-resourced settings [ 24 , 25 , 29 ]. As part of global investments to improve the quality of midwifery education and training, Liverpool School of Tropical Medicine (LSTM) in collaboration with the UNFPA Headquarters Global Midwifery Programme and Kenya midwifery educators developed a blended midwifery educator CPD programme (described in detail in the methods section). The CPD programme modules in this programme are aligned to the WHO’s midwifery educators’ core competencies [ 7 ] and ICM essential competencies for midwifery practice [ 14 ]. The programme is also aligned to the nursing and midwifery practice national regulatory requirements of Nursing and Midwifery Councils in LMICs such as Kenya and Nigeria, and relevant national policy [ 30 , 31 , 32 ].This programme aimed at sustaining and improving the educators’ competencies in delivery of their teaching, assessments, mentoring and feedback to students. To promote uptake, there is need to test the relevance and practicability of the CPD programme. Feasibility studies are used to determine whether an intervention is appropriate for further testing, relevant and sustainable in answering the question – Can it work [ 33 ]? The key focus of these studies are acceptability of the intervention, resources and ability to manage and implement intervention (availability, requirements, sustainability), practicality, adaptation, integration into the system, limited efficacy testing of the intervention in controlled settings and preliminary evaluation of participant responses to the intervention [ 33 , 34 , 35 ].

This study evaluated the feasibility of the LSTM/UNFPA midwifery educator CPD programme using the Kirkpatrick’s model for evaluating training programmes [ 36 ]. This model is an effective tool with four levels for evaluating training programmes. Level 1 (Participants’ reaction to the programme experience) helps to understand how satisfying, engaging and relevant participants find the experience. Level 2 (Learning) measures the changes in knowledge, skills and confidence after training. Level 3 (Behaviour) measures the degree to which participants apply what they learned during training when they are back on job and this can be immediately and several months after the training. This level is critical as it can also reveal where participants might need help to transfer learning during the training to practice afterwards. Level 4 (Results) measures the degree to which targeted outcomes occur because of training. In this study, participants’ reaction to the programme – satisfaction and relevance of the programme to meeting their needs (level 1) and change in knowledge, confidence and skills after the CPD programme (level 2) were assessed. Also, user perspectives and barriers to implementing the CPD programme were explored.

Study design

This was a mixed methods intervention study using a concurrent nested/embedded/convergent design conducted in Kenya and Nigeria in May and June 2023. This was designed to evaluate the feasibility of the midwifery educator CPD programme. The goal was to obtain different but complementary data to better understand the CPD programme with the data collected from the same participants or similar target populations [ 37 ].

The quantitative component of the evaluation used a quasi-experimental pre-post and post-test only designs to evaluate the effectiveness of the blended CPD programme intervention among midwifery educators from mid-level training colleges and universities from the two countries. Pre and post evaluation of knowledge (online self-directed component) and skills (developing a teaching plan during the face-to-face component) was performed. Post intervention evaluation on programme satisfaction, relevance of CPD programme and microteaching sessions for educators was conducted.

The qualitative component of the evaluation included open-ended written responses from the midwifery educators and master trainers to describe what worked well (enablers), challenges/barriers experienced in the blended programme and key recommendations for improvement were collected. In addition, key informant interviews with the key stakeholders (nursing and midwifery councils and the national heads of training institutions) were conducted. Data on challenges anticipated in the scale up of the programme and measures to promote sustainability, access and uptake of the programme were collected from both educators and key stakeholders.

A mixed methods design was used for its strengths in (i) collecting the two types of data (quantitative and qualitative) simultaneously, during a single data collection phase, (ii) provided the study with the advantages of both quantitative and qualitative data and (iii) helped gain perspectives and contextual experiences from the different types of data or from different levels (educators, master trainers, heads of training institutions and nursing and midwifery councils) within the study [ 38 , 39 ].

The study was conducted in Kenya and Nigeria. Kenya has over 121 mid-level training colleges and universities offering nursing and midwifery training while Nigeria has about 300. Due to the vastness in Nigeria, representative government-owned nursing and midwifery training institutions were randomly selected from each of the six geo-political zones in the country and the Federal Capital Territory. Mid-level training colleges offer the integrated nursing and midwifery training at diploma level while universities offer integrated nursing and midwifery training at bachelor/master degree level in the two countries (three universities in Kenya offer midwifery training at bachelor level). All nurse-midwives and midwives trained at both levels are expected to possess ICM competencies to care for the woman and newborn. Midwifery educators in Kenya and Nigeria are required to have at least advanced diploma qualifications although years of clinical experience are not specified.

It is a mandatory requirement of the Nursing and Midwifery Councils for nurse/midwives and midwifery educators in both countries to demonstrate evidence of CPD for renewal of practising license in both countries [ 40 , 41 ]. A minimum of 20 CPD points (equivalent to 20 credit hours) is recommended annually for Kenya and 60 credit hours for Nigeria every three years. However, there are no specific midwifery educator CPD that incorporated both face-to-face and online modes of delivery, available for Kenya and Nigeria and indeed for many countries in the region. Nursing and midwifery educators are registered and licensed to practice nursing and midwifery while those from other disciplines who teach in the midwifery programme are qualified in the content they teach.

Study sites

In Kenya, a set of two mid-level colleges (Nairobi and Kakamega Kenya Medical Training Colleges (KMTCs) and two universities (Nairobi and Moi Universities), based on the geographical distribution of the training institutions were identified as CPD Centres of Excellence (COEs)/hubs. In Nigeria, two midwifery schools (Centre of Excellence for Midwifery and Medical Education, College of Nursing and Midwifery, Illorin, Kwara State and Centre of Excellence for Midwifery and Medical Education, School of Nursing Gwagwalada, Abuja, FCT) were identified. These centres were equipped with teaching and EmONC training equipment for the practical components of the CPD programme. The centres were selected based on the availability of spacious training labs/classes specific for skills training and storage of equipment and an emergency obstetrics and newborn care (EmONC) master trainer among the educators in the institution. They were designated as host centres for the capacity strengthening of educators in EmONC and teaching skills.

Intervention

Nursing and midwifery educators accessed and completed 20 h of free, self-directed online modules on the WCEA portal and face-to-face practical sessions in the CPD centres of excellence.

The design of the midwifery educator CPD programme

The design of the CPD modules was informed by the existing gap for professional development for midwifery educators in Kenya and other LMICs and the need for regular updates in knowledge and skills competencies in delivery of teaching [ 9 , 15 , 23 , 28 ]. Liverpool School of Tropical Medicine led the overall design of the nursing and midwifery educator CPD programme (see Fig.  1 for summarised steps taken in the design of the blended programme).

This was a two-part blended programme with a 20-hour self-directed online learning component (accessible through the WCEA platform at no cost) and a 3-day face-to-face component designed to cover theoretical and practical skills components respectively. The 20-hour self-directed online component had four 5-hour modules on reflection practice, teaching/learning theories and methods, student assessments and effective feedback and mentoring. These modules had pretest and post-test questions and were interactive with short videos, short quizzes within modules, links for further directed reading and resources to promote active learning. This online component is also available on the WCEA platform as a resource for other nurses and midwifery educators across the globe ( https://wcea.education/2022/05/05/midwifery-educator-cpd-programme/ ).

Practical aspects of competency-based teaching pedagogy, clinical teaching skills including selected EmONC skills, giving effective feedback, applying digital innovations in teaching and learning for educators and critical thinking and appraisal were delivered through a 3-day residential face-to-face component in designated CPD centres of excellence. Specific skills included: planning and preparing teaching sessions (lesson plans), teaching practical skills methodologies (lecture, simulation, scenario and role plays), selected EmONC skills, managing teaching and learning sessions, assessing students, providing effective feedback and mentoring and use of online applications such as Mentimeter and Kahoot in formative classroom assessment of learning. Selected EmONC skills delivered were shoulder dystocia, breech delivery, assisted vaginal delivery (vacuum assisted birth), managing hypovolemic shock and pre-eclampsia/eclampsia and newborn resuscitation. These were designed to reinforce the competencies of educators in using contemporary teaching pedagogies. The goal was to combine theory and practical aspects of effective teaching as well as provide high quality, evidence-based learning environment and support for students in midwifery education [ 4 ]. These modules integrated the ICM essential competencies for midwifery practice to provide a high quality, evidence-based learning environment for midwifery students. The pre and post tests form part of the CPD programme as a standard assessment of the educators.

As part of the design, this programme was piloted among 60 midwifery educators and regulators from 16 countries across Africa at the UNFPA funded Alliance to Improve Midwifery Education (AIME) Africa regional workshop in Nairobi in November 2022. They accessed and completed the self-directed online modules on the WCEA platform, participated in selected practical sessions, self-evaluated the programme and provided useful feedback for strengthening the modules.

The Nursing and Midwifery Councils of Kenya and Nigeria host the online CPD courses from individual or organisation entities on the WCEA portal. In addition, the Nursing Council of Kenya provides opportunities for self-reporting for various CPD events including accredited online CPD activities/programmes, skill development workshops, attending conferences and seminars, in-service short courses, practice-based research projects (as learner, principal investigator, principal author, or co-author) among others. In Nigeria, a certificate of attendance for Mandatory Continuing Professional Development Programme (MCPDP) is required as evidence for CPD during license renewal. However, the accredited CPD programmes specific for midwifery educators are not available in both countries and Africa region [ 15 , 42 ].

Midwifery educator CPD programme design stages

Participants and sample size

Bowen and colleagues suggest that many feasibility studies are designed to test an intervention in a limited way and such tests may be conducted in a convenience sample, with intermediate rather than final outcomes, with shorter follow-up periods, or with limited statistical power [ 34 ].

A convenience random sample across the two countries was used. Sample size calculations were performed using the formula for estimation of a proportion: a 95% confidence interval for estimation of a proportion can be estimated using the formula: \(p\pm 1.96\sqrt{\frac{\text{p}(1-\text{p})}{n}}\) The margin of error (d) is the second term in the equation. For calculation of the percentage change in competence detectable Stata’s power paired proportion function was used.

To achieve the desired level of low margin of error of 5% and a 90% power (value of proportion) to detect competence change after the training, a sample of 120 participants was required. Using the same sample to assess competence before and after training, so that the improvement in percentage competent can be derived and 2.5% are assessed as competent prior to training but not after training (regress), a 90% power would give a 12% improvement change in competence after the training.

A random sample of 120 educators (60 each from Kenya & Nigeria; 30 each from mid-level training colleges and universities) were invited to participate via an email invitation in the two components of the CPD programme (Fig.  2 ). Importantly, only participants who completed the self-directed online modules were eligible to progress to the face-to-face practical component.

Flow of participants in the CPD programme (SDL = self-directed online learning; F2F = face-to-face practical)

For qualitative interviews, eight key informant interviews were planned with a representative each from the Nursing and Midwifery Councils, mid-level training institutions’ management, university and midwifery associations in both countries. Interviews obtained data related to challenges anticipated in the scale up of the programme and measures to promote sustainability, access and uptake of the programme.

Participant recruitment

Only nursing and midwifery educators registered and licensed by the Nursing and Midwifery Councils were eligible and participated. This was because they can access the WCEA website with the self-directed online programme via the Nursing and Midwifery Councils’ websites, only accessible to registered and licensed nurses and midwives.

The recruitment process was facilitated through the central college management headquarters (for mid-level training colleges’ educators) and Nursing and Midwifery Councils (for university participants). Training institutions’ heads of nursing and midwifery departments were requested to share the contact details of all educators teaching midwifery modules, particularly the antepartum, intrapartum, postpartum and newborn care modules in the two countries. A list of 166 midwifery educators from 81 universities and mid-level training colleges was obtained through the Heads of the Department in the institutions.

The research lead, with the assistance by the co-investigator from Nigeria then randomly sampled 120 educators based on institution type and region for representativeness across the countries. Following the selection of participants, the two investigators shared the electronic detailed participant study information sheet and consent form to the potential participants one week before the start of the self-directed online modules. Clear guidance and emphasis on the conduct of the two-part program including completing the mandatory four self-directed online modules was provided. Due to the large number of eligible participants, the recruitment and consenting process was closed after reaching the first 30 participants consenting per institution type and region, with 1–2 educators per institution randomly recruited. This allowed as many institutions to be represented across the country as possible. Participants received a study information sheet and an auto-generated copy of the electronic consent form completed in their emails. Other opportunities for participating in the two-part programme were provided as appropriate for those who missed out. Only those who completed the four online modules were invited for the practical component. A WhatsApp community group for the recruited participants was formed for clarifications about the study, troubleshooting on challenges with online access and completion of the modules before and during the programme.

Self-directed online component

Upon consenting, the contact details of the educators from each level were shared with WCEA program director for generation of a unique identification code to access the self-directed online modules on the WCEA portal. Educators completed their baseline characteristics (demographic and academic) in the online platform just before the modules. Each self-directed online module was estimated to be completed in five hours. Only after completing a module was the participant allowed to progress to the next module. The modules were available for participants to complete at their own time/schedule. An autogenerated certificate of completion with the participant’s post-completion score was awarded as evidence of completing a module. Participants completed a set of 20 similar pretest and posttest multiple choice questions in each module for knowledge check. A dedicated staff from WCEA actively provided technical support for educators to register, access and complete the online modules. At the end of each module, participants completed a self-evaluation on a 5-point Likert scale for satisfaction (0 = very unsatisfied, 1 = unsatisfied, 2 = neutral, 3 = satisfied and 4 = very satisfied) and relevance of the modules (0 = very irrelevant, 1 = irrelevant, 2 = neutral, 3 = relevant and 4 = very relevant). This provided participants’ reactions to the different components of the modules on whether they met the individual educator’s development needs. In addition, participants responded to the open-ended questions at the end of the modules. These were on what they liked about the modules, challenges encountered in completing the modules and suggestions for improvement of the modules. A maximum period of two weeks was given for educators to complete the modules before progressing to the practical component.

Practical component

The practical component was delivered by a pool of 18 master trainers who received a 1-day orientation from the research lead before the training. The master trainers were a blend of experienced midwifery and obstetrics faculty in teaching and clinical practice actively engaged in facilitating EmONC trainings selected from Kenya and Nigeria. Four of these master trainers from Kenya participated in the delivery of both sets of trainings in Kenya and Nigeria.

Only educator participants who completed the self-directed online modules and certified were invited to participate in a 3-day residential practical component. Two separate classes were trained (mid-level and university level educators) per country by the same group of eight master trainers. The sessions were delivered through short interactive lectures; small group and plenary discussions; skills demonstrations/simulations and scenario teaching in small breakout groups; role plays and debrief sessions. Sessions on digital innovations in teaching and learning were live practical sessions with every participant using own laptop. Nursing and Midwifery Councils representatives and training institutions’ managers were invited to participate in both components of the programme.

Participant costs for participating in the two-part CPD programme were fully sponsored by the study. These were internet data for completing the self-directed online component and residential costs – transport, accommodation, and meals during the practical component.

Data collection

Self-directed online knowledge pretests and post-tests results, self-rated measures of satisfaction and relevance of the modules including what they liked about the modules, challenges encountered in accessing and completing the modules and suggestions for improvement data was extracted from the WCEA platform in Microsoft Excel.

On day 1 of the practical component, participants using their personal computers developed a teaching plan. On the last day (day 3), participants prepared a teaching plan and powerpoint presentation for the microteaching sessions. No teaching plan template from the trainers was provided to the participants before the training. However, they used formats from their institutions if available. A standard teaching plan template was provided at the end of the training.

The group of master trainers and participants were divided into groups for the microteaching sessions which formed part of the formative assessment. Each participant delivered a powerpoint presentation on a topic of interest (covered in the teaching plan) to the small group of 13–15 participants. This was followed by a structured session of constructive feedback that started with a self-reflection and assessment. This was followed by peer supportive and constructive feedback from the audience participants and faculty/master trainers identifying areas of effective practice and opportunities for further development. Each microteaching session lasted 10–15 min. Each of the microteaching session presentation and teaching plan were evaluated against a pre-determined electronic checklist by two designated faculty members independently during/immediately after the microteaching session. The checklist was adapted from LSTM’s microteaching assessment of the United Kingdom’s Higher Education Academy (HEA)’s Leading in Global Health Teaching (LIGHT) programme. The evaluation included preparing a teaching plan, managing a teaching and learning session using multiple interactive activities, designing and conducting formative assessments for learning using digital/online platforms, and giving effective feedback and critical appraisal. The master trainers received an orientation training on the scoring checklist by the lead researcher/corresponding author.

Self-rated confidence in different teaching pedagogy skills were evaluated before (on day 1) and after (day 3) the training on a 5-point Likert scale (0 = not at all confident, 1 = slightly confident, 2 = somewhat confident, 3 = quite confident and 4 = very confident). A satisfaction and relevance of practical component evaluation on a 5-point Likert scale was completed by the participants on an online designed form on day 3 after the microteaching sessions of the practical component. This form also had a similar qualitative survey with open-ended questions on what they liked about the practical component, challenges encountered in completing the practical component and suggestions for improvement of the component.

Using a semi-structured interview guide, six qualitative key informant interviews, each lasting about 30–45 min, were conducted by the lead researcher with the Nursing and Midwifery Councils focal persons and training institutions’ managers. These were audio recorded in English, anonymized, and deleted after transcription. These interviews were aimed at getting their perspectives on the programme design, anticipated barriers/enablers with the CPD programme and strategies for promoting uptake of the CPD programme. These interviews were considered adequate due to their information power (indicating that the more information the sample holds, relevant for the actual study, the lower amount of participants is needed) [ 43 ] and upon obtaining data saturation, considered the cornerstone of rigor in qualitative research [ 44 , 45 ].

Assessment of outcomes

Participants’ reaction to the programme (satisfaction and relevance) (Kirkpatrick level 1) was tested using the self-rated 5-point Likert scales. Change in knowledge, confidence and skills (Kirkpatrick level 2) was tested as follows: knowledge through 20 pretest and post-test multiple choice questions per module in the self-directed online modules; confidence in applying different pedagogy skills through the self-rated 5-point Likert scale; and teaching skills through the observed microteaching sessions using a checklist.

Reliability and validity of the data collection tools

The internal consistency (a measure of the reliability, generalizability or reproducibility of a test) of the Likert scales/tools assessing the relevance of the online and practical modules and satisfaction of educators with the two blended modules were tested using the Cronbach’s alpha statistic. The Cronbach’s alpha statistics for the four Likert scales/tools ranged from 0.835 to 0.928, all indicating acceptably good to excellent level of reliability [ 46 ]. Validity (which refers to the accuracy of a measure) of the Likert scales were tested using the Pearson correlation coefficient statistic. Obtained correlation values were compared to the critical values and p-values reported at 95% confidence intervals. All the scales were valid with obtained Pearson correlation coefficients reported − 0.1946, which were all greater than the critical values ( p  < 0.001) [ 46 ]. The semi-structured interview guides for the qualitative interviews with the training institutions’ managers and midwifery councils (regulators) were developed and reviewed by expert study team members with experience in qualitative research.

Data management and analysis

Data from the online/electronic tools was extracted in Microsoft Excel and exported to SPSS version 28 for cleaning and analysis. Normality of data was tested using the Kolmogorov-Smirnov test suitable for samples above 50. Proportions of educator characteristics in the two countries were calculated. Differences between the educator characteristics in the two countries were tested using chi-square tests (and Fishers-exact test for cells with counts of less than 5).

For self-rated relevance of CPD programme components and satisfaction with the programme on the 0–4 Likert scales, descriptive statistics were calculated (median scores and proportions). Results are presented as bar graphs and tables. Cronbach alpha and Pearson correlation coefficients were used to test the reliability and validity of the test items respectively.

Change in knowledge in online modules, confidence in pedagogy skills and preparing teaching plans among educators was assessed by comparing pre-training scores and post-training scores. Descriptive statistics are reported based on normality of data. Differences in the scores were analysed using the Wilcoxon signed ranks tests, a non-parametric equivalent of the paired t-test. Differences between educators scores in microteaching by country and institution type were performed by Mann-Whitney U test. Level of competence demonstrated in the teaching plan and microteaching skill was defined as the percentage of the desired characteristics present in the teaching plan and microteaching session, set at 75% and above. The proportion of participants that achieved the desired level of competence in their teaching plan and microteaching skill was calculated. Binary logistic regression models were used to assess for the strengths of associations between individual educator and institutional characteristics (age, gender, qualifications, length of time as educator, training institution and country) and the overall dichotomised competent score (proportion achieved competence in teaching plan and microteaching skills). P-values less than 0.05 at 95% confidence interval were considered statistically significant.

Preparation for qualitative data analysis involved a rigorous process of transcription of recorded interviews with key informants. In addition, online free text responses by midwifery educators on what worked well, challenges encountered, and recommendations were extracted in Microsoft Excel format and exported to Microsoft Word for data reduction (coding) and theme development. Qualitative data was analysed using thematic framework analysis by Braun and Clarke (2006) as it provides clear steps to follow, is flexible and uses a very structured process and enables transparency and team working [ 47 ]. Due to the small number of transcripts, computer assisted coding in Microsoft Word using the margin and comments tool were used. The six steps by Braun and Clarke in thematic analysis were conducted: (i) familiarising oneself with the data through transcription and reading transcripts, looking for recurring issues/inconsistencies and, identifying possible categories and sub-categories of data; (ii) generating initial codes – both deductive (using topic guides/research questions) and inductive coding (recurrent views, phrases, patterns from the data) was conducted for transparency; (iii) searching for themes by collating initial codes into potential sub-themes/themes; (iv) reviewing themes by generating a thematic map (code book) of the analysis; (v) defining and naming themes (ongoing analysis to refine the specifics of each sub-theme/theme, and the overall story the analysis tells); and (vi) writing findings/producing a report. Confidentiality was maintained by using pseudonyms for participant identification in the study. Trustworthiness was achieved by (i) respondent validation/check during the interviews for accurate data interpretation; (ii) using a criterion for thematic analysis; (iii) returning to the data repeatedly to check for accuracy in interpretation; (iv) quality checks and discussions with the study team with expertise in mixed methods research [ 39 , 47 ].

Integration of findings used the parallel-databases variant and are synthesised in the discussion section. In this common approach, two parallel strands of data are collected and analysed independently and are only brought together during interpretation. The two sets of independent results are then synthesized or compared during the discussion [ 39 ].

Quantitative findings

Midwifery educators’ characteristics.

A total of 116 (96.7%) and 108 (90.0%) educators from 81 institutions completed the self-directed online learning and practical component respectively from the two countries. There were no significant differences between countries in educators’ qualifications, when last taught a midwifery class and whether attended any CPD training in the preceding year before the study ( p  > 0.05). Overall, only 28.7% of the educators had a midwifery related CPD training in the preceding year before the study. Midwifery educator characteristics are outlined below (Table  1 ).

Change in knowledge

This was assessed in each of the four self-directed online modules. The results from ranked scores based on Wilcoxon signed ranks test showed significant improvements in educators’ knowledge in all the four online modules completed ( p  < 0.001). The highest mean score improvement was observed in students’ assessment module, 48.1% (SD ± 15.1) to 85.2% (SD ± 15.7), a 37.1% improvement. Improvements in knowledge in the other modules were as follows: reflective practice (27.6%), mentoring and giving effective feedback (27.4%) and teaching methods (19.2%). Overall knowledge score for all modules improved from 52.4% (SD ± 10.4) to 80.4 (SD ± 8.1), p  < 0.001 (Table  2 ).

Relevance of self-directed online modules

The internal consistency of each of the four modules was tested with Cronbach’s alpha. The overall Cronbach’s alpha for the four items was 0.837, a good and acceptable level of reliability. All the four modules assessed were valid with calculated Pearson correlation coefficient values greater than the critical value of 0.1946 ( p  < 0.001) at 95% confidence interval.

Educators from the two countries, on a scale of 0–4 rated the online modules as very relevant with a median score of 4 out of 4 (IQR 0) for each of the four modules: reflective practice, teaching methods, students’ assessments and mentoring and giving effective feedback. There were no ratings of 0, 1 and 2 for all the modules (Fig.  3 ).

Educators’ ratings of the relevance of self-directed online modules

Satisfaction with the self-directed online modules

The internal consistency of each of the eight items was tested with Cronbach’s alpha. The overall Cronbach’s alpha for the eight items was 0.928, an excellent level of reliability. All the eight items assessed were valid with their obtained Pearson correlation coefficient values greater than the critical value of 0.1946 ( p  < 0.001) at 95% confidence interval.

Each of the eight items rated on satisfaction had a median score of 4 out of 4 (IQR 0). Over 80% of the educators were very satisfied with the online modules’ content as presented in a logical format and informative. Also, the modules helped them to learn something new, updated their knowledge and the materials were useful and valuable for their practice. Over 70% were very satisfied with the modules as they helped them refresh their knowledge and skills with the links and activities embedded in the modules useful in adding to their learning. None of the educators were dissatisfied (rated 0 or 1) with the online modules (Table  3 ).

Change in confidence in different pedagogy skills

The internal consistency of each of the eight items assessed was tested with Cronbach’s alpha using the baseline data. The overall Cronbach’s alpha for the eight items was 0.893, a good level of reliability. All the eight items assessed were valid with their obtained Pearson correlation coefficient values greater than the critical value of 0.1946 ( p  < 0.001) at 95% confidence interval.

Changes in confidence before and after the training were compared using the Wilcoxon signed rank test, a parametric equivalent of the paired t-test when data is not normally distributed. The mean score of self-rated confidence of educators on a scale of 0–4 for all the eight skills significantly improved after the training from 2.73 (SD ± 0.68) to 3.74 (SD ± 0.34) ( p  < 0.001). Mean confidence was highest in facilitating a lecture (3.23, SD ± 0.8) and lowest on using digital innovations (Mentimeter) in formative assessment of teaching/learning (1.75, SD ± 1.15) before the training. These improved significantly after the training to 3.84 (SD ± 0.41) for facilitating a lecture and 3.50 (SD ± 0.63) for using digital innovations (Mentimeter) in formative assessment of teaching/learning, p  < 0.001. The mean confidence of educators was largely average before the training and significantly improved after the training in six skills ( p  < 0.001). These were designing learning outcomes using measurable Bloom’s taxonomy verbs, preparing a teaching plan, identifying relevant resources to enhance learning, facilitating a scenario teaching, facilitating a practical simulation/demonstration and giving effective feedback for learning (Table  4 ).

Preparing a teaching plan and microteaching skills

The overall median score in preparing a teaching plan was 63.6% (IQR 45.5) before the training and improved significantly to 81.8% (IQR 27.3) after the training, p  < 0.001. The median scores differed significantly by country before and after the training. Before the training, Kenyan educators had higher median scores (72.7%, IQR 27.3) compared to Nigeria counterparts (54.5%, IQR 36.4), p  < 0.001. After the training, Kenyan educators had significantly higher median scores (81.2%, IQR 18.2) than Nigerian counterparts (72.7%, IQR 18.2), p  = 0.024. However, there were no significant differences in the median scores between the training institutions before and after the training, p  > 0.05. For microteaching, the overall median score was 76.5% (IQR 29.4). There were no significant differences between countries and training institutions in the microteaching scores, p  > 0.05. Kenya educators (82.4%, IQR 29.4) had slightly higher scores than Nigeria (76.5%, IQR 29.4), p  = 0.78. Mid-level educators (79.4%, IQR 29.4) had slightly higher scores than university educators (76.5%, IQR 28.7), p  = 0.515 (Table  5 ).

The inter-rater reliability/agreement of the eight pairs of assessors in both countries were assessed by Cohen Kappa statistic. The Kappa statistics for the eight pairs ranged between 0.806 and 0.917, p  < 0.001, showing near perfect agreement between the pairs of assessors.

Association between independent educator and institutional characteristics and the microteaching skill scores

Categorised skills scores (≥ 75% mean score as competent) showed that 55 (51.4%) and 62 (57.9%) of the educators scored 75% or higher in the teaching plan preparation and microteaching skill assessments respectively. Logistic regression analysis showed that educator’s country, age, gender, qualifications, training institution type and length as educator were not significantly associated with the overall categorised teaching plan or microteaching scores ( p  > 0.05).

Relevance of the practical component

The internal consistency of each of the six skills items was tested with Cronbach’s alpha. The overall Cronbach’s alpha for the six items was 0.866, a good level of reliability. All the six skills items assessed were valid with their obtained Pearson correlation coefficient values greater than the critical value of 0.1946 ( p  < 0.001) at 95% confidence interval.

On a self-rating Likert scale of 0–4, the median score for each of the six skills assessed and trained was 4 out of a maximum of 4, indicating that the educators found the different pedagogy skills very relevant after the training. Over 80% of the educators rated the sessions on teaching plan (85.2%), scenario teaching (87.0%), simulation/demonstration teaching (82.4%) and giving effective feedback (85.2%) as very relevant. Over three-quarters (77.8%) of the educators rated the sessions on lecture teaching and use of digital innovations (Mentimeter) in assessment as very relevant (Fig.  4 ).

Relevance of the practical components

Satisfaction with the practical component

The internal consistency of each of the six skills items was tested with Cronbach’s alpha. The overall Cronbach’s alpha for the six items was 0.835, a good level of reliability. All the six skills items assessed were valid with their obtained Pearson correlation coefficient values greater than the critical value of 0.1946 ( p  < 0.001) at 95% confidence interval.

On a self-rating Likert scale of 0–4, the median score for each of the six skills assessed was 4 out of a maximum of 4, indicating that educators were very satisfied with the practical skills sessions. Over 70% of the educators were very satisfied with the sessions on giving effective feedback (79.6%), lecture teaching (75.9%), scenario and simulation teaching (73.1% each). Two-thirds of the educators (67.6%) were very satisfied with the digital innovations in teaching (use of Mentimeter) for formative assessment in teaching and learning. All educators were satisfied with the preparing of teaching plan in teaching and learning with the majority (63.0%) as very satisfied while the remaining 37.0% satisfied. None of the educators were dissatisfied with the practical component of the training (Fig.  5 ).

Satisfaction with practical skills

Qualitative findings

What educators liked about the self-directed online modules.

Educators from both levels and countries had similar views on the online component. These are broadly summarised under the sub-themes: (i) educative and relevant for practice, (ii) flexible and convenient learning and (iii) motivating, interesting and interactive.

Educative and relevant for practice

Educators reported the online modules as educative and informative and, improved their knowledge in teaching, assessments, reflective practice and providing effective feedback to students to promote learning as well as increasing their self-confidence and critical thinking skills. Besides, educators found the modules valuable and relevant for their professional growth and practice.

“The modules were well organized, they were relevant to my practice and met my expectations” university midwifery educator, Kenya. “The materials are very rich with current information to guide. Very informative & valuable to my professional growth” university midwifery educator, Nigeria.

Flexible and convenient learning

Educators reported that they could access and complete the online modules at their flexible and convenient time. This flexibility enhanced and stimulated them to complete the informative modules at their comfort times either at home or office without disruption to their schedules.

“(The modules) gave me ample time to read at my own pace and time without any hurry to understand the content well. They were well organised. Also, flexibility of learning and the access to materials was excellent” university midwifery educator, Kenya. “It is flexible and convenient. It empowers the learner to take ownership of the learning process. Learning is personalized” mid-level training college midwifery educator, Nigeria.

Motivating, interesting and interactive

Educators reported that the online modules were well structured, motivating, interesting and had components that promoted interaction for learning. For example, pretests, various quizzes within the modules and posttest questions and the added specific short extra reading segments promoted interaction and learning.

“The intermittent assessment questions. It helped maintain my focus” university midwifery educator, Nigeria . “Very interactive. They were very informative and extra reading assignments complemented the content” university midwifery educator, Kenya .

Challenges encountered with the self-directed online learning modules

Four sub-themes emerged that summarised the challenges experienced by midwifery educators in the two countries to access and complete the self-directed online modules. These are (i) network/internet connectivity, (ii) technology challenges, (iii) electricity power supply and power outages and, (iv) time constraints.

Network/internet connectivity

Network and internet connectivity difficulties and fluctuations was the commonest reported challenge in completing the self-directed online modules by educators from both countries. This affected the access, progress, downloading extra resources embedded within the modules and completing the integrated evaluations within the modules.

“Accessing the modules, problem with submitting forms and exams, had network problem” mid-level training college midwifery educator, Nigeria . “I kept going offline and I would have to restart every time. They were too internet dependent” university midwifery educator, Kenya.

Technology challenges

Technological challenges were observed as well as reported among educators from both countries. These ranged from poor access to emails due to forgotten email addresses, usernames or passwords, difficult access and navigation through the online modules, completing the matching questions that required dragging items, completing the evaluations and downloading certificates after completion of the modules.

“I am not very good with ICT, so I had issues using my laptop” mid-level training college midwifery educator, Nigeria. “Accessibility was difficult. I had to restart the process a number of times. The modules would sometimes take you back more than 20 slides which delayed the completion rate” university midwifery educator, Kenya.

Electricity power supply interruptions and fluctuations

Power interruptions, fluctuations and outages especially in Nigeria were cited as a challenge to complete the online modules. This delayed the completion of the modules as electric power was critical to access and complete the modules on either WCEA app on mobile phones or computers.

“The modules should not start from beginning whenever there is interrupted power supply” MLC midwifery educator, Nigeria. “Network failure due to interrupted power supply” university midwifery educator, Nigeria.

Time constraints

Although educators commented the flexibility with which to complete the online modules, time to complete the online modules was also cited as a challenge in both countries.

“It requires a lot of time, this is a challenge because I am also involved with other activities at the place of work which require my attention” university midwifery educator, Kenya.

What educators liked about the practical component

Educators written feedback on what they liked about the practical component of the CPD programme was categorised into the four sub-themes: new knowledge and relevant for practice; improved knowledge, skills and confidence to teach; enhanced participatory and active learning; individualised support in learning.

New knowledge and relevant for practice

The practical component provided new learning particularly on the use of digital platforms (Mentimeter and Kahoot) for formative assessment to evaluate learning during classroom teaching. In their integrated teaching using both online and face-to-face delivery, use of technology (Mentimeter and Kahoot) in classroom assessment was not a common practice as most of them had not heard about the available online platforms. They found Mentimeter (and Kahoot) to be interesting resources for formative assessments in class to facilitate teaching and learning. The techniques of giving effective feedback using the sandwich and ‘stop, start, continue’ methods were viewed to promote interaction between the educator and the learner for effective learning. Educators also acknowledged new knowledge and skills updates on EmONC relevant for their practice.

“Giving feedback, innovation of the online formative assessment, the teaching plan. I wish we would adapt them for daily application rather than the traditional teacher centered one.” Mid-level training college educator, Kenya . “(I liked) Everything, especially the technological innovations for assessment” Mid-level training college educator, Nigeria .

Improved knowledge, skills and confidence to teach

Educators reported that the practical sessions were interactive and engaging with good combination of theory and practice which facilitated learning. They reported that participating in the practical component enabled them to update and improve their knowledge, skills and confidence in planning and delivering theoretical and practical teaching using multiple methods. Similar improvements were reported on preparing and conducting students’ assessments and giving effective feedback to promote learning. On use of technology in formative assessments, the interactive practical sessions boosted the confidence of educators in using Mentimeter (and Kahoot) online platforms during classroom teaching.

“It helped build my confidence, had hands on practice on clinical skills and teaching skills, learnt about outdated practices and current evidence based clinical and teaching skills.” Mid-level training college educator, Nigeria . “They were very interesting especially the scenarios and skills. I was able to enhance my practical skills and technology in evaluating learning.” University midwifery educator, Kenya .

Enhanced participatory and active learning

The practical component complemented the self-directed online learning for educators. They highly commented and benefitted from the hands-on opportunities to actively engage through return demonstrations during the practical programme. This component also enabled them to brainstorm and contribute actively during the sessions. They highlighted that the practical component enhanced and reinforced learning through active participation in demonstrations, questions, group discussions and plenary sessions.

“This face-to-face module provided me with the opportunity to brainstorm with other educators, facilitators and resource persons. This will enhance my teaching skills.” Mid-level training college midwifery educator, Nigeria . “Interaction with facilitators who could clarify points that I had earlier not understood, interaction with other participants and was also able to learn from them.” University midwifery educator, Kenya .

Individualised support in learning

Educators received individualised peer support and learning during the practical component. They had opportunities within the small breakout groups for peer learning and one-to-one support from the facilitators to update and learn new knowledge and skills.

“A chance to get immediate feedback was availed by the presenters.” University midwifery educator, Kenya . “Facilitators were well informed and gave learners opportunity for return demonstration and support.” Mid-level training college midwifery educator, Kenya .

Challenges encountered with the practical component

Key challenges reported by the mixed group of educators and master trainers across the two countries include: inadequate time, computer technology challenges and poor internet connectivity for practical components.

Inadequate time

Although small breakout sessions were utilised to provide each educator with an opportunity to practice the skills, it was commonly reported that time was inadequate for skills demonstrations and return demonstrations by all educators. This was especially for areas educators had inadequate knowledge and new skills that were observed thus adequate time for teaching and repeat demonstrations for mastery was required. Similar observations were made by the master trainers who felt that some educators had never encountered some of the basic EmONC skills demonstrated or never practised and thus required a longer duration for familiarisation and practice.

“Time was short hence not enough to return demo” Mid-level training college midwifery educator, Kenya . “Some of the things were new and required more time for demonstration and practice.” Mid-level training college midwifery educator, Nigeria .

Computer technology challenges and poor internet connectivity for practical components

Some educators encountered technical difficulties in using computers during the practical component. In some cases, this was compounded by poor network/internet connectivity. This delayed completion of practical components requiring the use of computers including pretests, preparing teaching plans and presentations, post-tests and classroom demonstrations using digital innovations in teaching and learning. However, assistance was provided by the trainers as appropriate to those who needed technical support.

“(There were) technical challenges with use of computers for few participants.” Master trainer, Nigeria . “Slow internet can hinder smooth flow of sessions.” Master trainer, Kenya .

Key areas for additional support

For quality education and training, master trainers generally recommended that all educators should be trained and regularly supported in the basic EmONC course to strengthen their competencies for effective teaching of EmONC skills. Further support in computer technology use including basics in navigation around windows/programmes, formatting in Microsoft Office Word and Powerpoint, literature searching, and referencing were other critical components to be strengthened.

Perspectives from training institutions managers and midwifery regulators

Measures to ensure midwifery educators take specific cpds that have been designed to improve their teaching competencies.

Key informant interviews with the pre-service training institutions’ managers and nursing and midwifery councils from the two countries were conducted and revealed key strategies outlined below that should ensure access and completion of the blended CPD programme specific for educators’ teaching competencies.

Awareness creation, integrating programme into policy and performance appraisal

The aspect of online CPD was highlighted as a new concept in Nigeria. Due to this novelty, the country was reluctant to accredit many online CPD programmes for in-service and pre-service nursing and midwifery personnel. However, the regulatory Nursing and Midwifery Council of Nigeria had established monitoring mechanisms to evaluate its uptake to meet the definition of CPD and is still work in progress.

“For the online, it’s actually a relatively new concept, in fact because of monitoring and evaluation, we have struggled with accepting online CPDs… So, we’re struggling on how to develop a guideline for online CPDs. So, we’re now starting with the WCEA. So far, only the WCEA has that approval to provide CPD…We said let’s look at how this works out before we can extend it to other providers.” Nursing and Midwifery Council, Nigeria .

Both countries emphasized the need to create awareness of the CPD programme for midwifery educators and a policy framework for CPD. Regulators emphasized the need to have the CPD programme as mandatory for all midwifery educators through a policy directive. They suggested that the blended CPD programme should form a mandatory specified proportion of the content addressing their specific competencies. Besides, the training institution recommended that the programme should form part of the educator’s performance appraisal on a regular basis. Active monitoring systems were suggested to be in place to ensure compliance of participation and completion to acquire specific relevant competencies in pedagogy.

“…Ensure that educators take the particular modules before license renewal. Tie modules that are related to midwifery education to the educators and make them mandatory. Yes, we make it as a matter of policy that you should be taking these courses over and over again.” Nursing and Midwifery Council, Nigeria .

It was strongly suggested that attaching incentives as motivators to completing the programme would attract educators to complete the CPD programme. These incentives include certification, recognition for participation in curriculum reviews, national examination setting, facilitating national examinations, promotion and service and eligibility as trainers of trainers to colleagues.

“You attach a course, one training, you cannot guarantee that these courses will be taken. So we find a way to attach something to it. You must have evidence that you attended these programs. So once you attach something like that, they will all flock because there is an incentive to it. Because we say, as an educator, before you go after every examination to examine students, you must have taken these courses.” Nursing and Midwifery Council, Nigeria .

Internet connectivity

Training institutions’ managers suggested investments in internet connectivity for training institutions to support educators access and complete the self-directed online programme. This was also highlighted as a critical challenge for the online component by the educators in both countries.

“The issues of internet connectivity and I think we need to be proactive about it so that we have a way to constantly bring it to the forefront especially in our policies. But connectivity would be a major area to look at as people are using their money.” Mid-level training college manager, Kenya .

Anticipated challenges in the scale-up of the CPD programme

Key challenges anticipated in the roll-out and scale-up of the blended CPD programme were identified as inadequate skills of the educators in the use of information and communication technology during the practical component (including preparation of powerpoint presentations and completing tasks using a computer), and participant costs to attend the practical component (including participants’ residential costs and investments in proctor technology for ensuring academic integrity and monitoring and evaluation tool for educators’ compliance.) It was also emphasized that due to low remuneration of the educators, additional costs from their pocket to undertake the CPD could be a limiting factor for the intended faculty development initiatives. Other challenges included maintaining quality and academic integrity of the programme, potential bias in the selection of educators to attend future CPD programmes that is based on pre-existing relationships and ensuring an adequate pool of in-country trainers of trainers with midwifery competencies to deliver the practical component of the CPD programme.

There were strong suggestions that personal commitment by educators was required for personal and professional development. There were observations that educators sometimes completed the professional development programmes purely for relicensing and not necessarily for professional development. Regulators and institutional managers emphasized that educators need to understand the value of continuous professional development and create time to participate in the targeted CPD programmes to improve their competencies.

“We do advise our nurses, or we continue to inform them that taking these courses shouldn’t be tied to license renewal. It shouldn’t be tied to licence expiration or renewal of licences. You should continue to take these courses to develop yourself and not waiting until your licence expired before you take the courses. Yes, we actually try as much as possible to dissociate the renewal of licences with these courses.” Nursing and Midwifery Council, Nigeria .

Key results

Our study evaluated the feasibility of what the authors believe to be the first blended programme with online and face-to-face learning available in Africa, as a tool to reach midwifery educators in both urban and rural low-resource areas. In addition, our study is in line to an important call by WHO, UNFPA, UNICEF and ICM for an effective midwifery educator with formal preparation for teaching and engages in ongoing development as a midwifery practitioner, teacher/lecturer and leader [ 6 , 7 ]. Consequently, our intervention is part of investments for improving and strengthening the capacity of midwifery educators for quality and competent midwifery workforce as recommended by multiple global reports [ 4 , 5 , 11 ] and other publications [ 12 , 15 , 23 , 42 ]. Our study findings showed that the midwifery educators were very satisfied with the blended CPD programme. Educators rated the programme as highly relevant, educative, flexible, interesting and interactive, improved their knowledge, confidence and practical skills in their professional competencies for practice. Use of digital technology in teaching and students’ assessment was found to be an effective and innovative approach in facilitating teaching and learning. Key challenges experienced by educators included deficiencies in computer technology use, internet/network connectivity for online components, time constraints to complete the blended programme and isolated electric power outages and fluctuations which affected completion of the self-directed online components. Costs for participating and completing the programme, motivation, investments in information and communication technology, quality assurance and academic integrity were highlighted as critical components for the scale-up of the programme by institutional managers and training regulators. Establishment of a policy framework for educators to complete mandatory specific and relevant CPD was recommended for a successful roll-out in the countries.

Interpretation of our findings

Our study findings demonstrated that educators found the theoretical and practical content educative, informative and relevant to their practice. Recent evidence showed that midwifery educators had no/limited connection with clinical practice or opportunities for updating their knowledge or skills [ 15 , 42 ]. This underscores the value and importance of regular opportunities of CPD specific for educators to improve their professional competencies. It has provided these educators with a flexible educational model that allows them to continue working while developing their professional practice.

The use of a blended programme was beneficial as educators’ needs were met. It provided opportunities for educators to reflect, critically think, internalise and complement what was learned in the self-directed online component during the practical phase. This approach has been considered a means to adequately prepare midwifery faculty and improving national midwifery programmes in low-resource and remote settings [ 48 , 49 ]. Use of self-directed online platforms has emerged as a key strategy to improve access to CPD with flexibility and convenience as educators take responsibility for their own learning. Evidence suggests that the flexibility of net-based learning offers the midwifery educators a new and effective educational opportunity that they previously did not have [ 50 , 51 ]. A practical – based learning is important in pre-service education settings where the capacity of midwifery educators needs to be strengthened [ 52 , 53 ]. However, without continuous regular training, the midwives’ competence deteriorate and this in turn threaten the quality of pre-service midwifery education [ 52 , 54 ]. Implementation of this flexible blended educational model allows educators to continue working while developing their professional practice.

The quality of educators is an important factor affecting the quality of graduates from midwifery programmes to provide quality maternal and newborn health services [ 7 ]. Evidence suggests that many midwifery educators are more confident with theoretical classroom teaching than clinical practice teaching and that they also struggle to maintain their own midwifery clinical skills [ 4 , 5 ]. Our findings showed that the programme was effective, and educators improved their knowledge, confidence and skills in teaching, students’ assessment, effective feedback, reflective practice, mentoring and use of digital innovations in teaching and assessments. Our findings are similar to other related models of capacity building midwifery educators in other developing countries [ 24 , 50 , 53 , 55 , 56 , 57 ]. It is expected that educators will apply the learning in their planning for teaching, delivery of interactive and stimulatory teaching, monitoring learning through formative and summative assessments and mentoring their students into competent midwives. This is a pathway for accelerating the achievement of maternal and newborn health SDGs, universal health coverage, ending preventable maternal mortalities and every newborn action plan targets.

The value for CPD on educators’ knowledge, confidence and skills has been demonstrated with opportunities for improvement. Specific CPD targeted to relevant professional competencies is beneficial to the profession, quality of graduates for maternal and newborn health care and global targets. However, further investments in strengthening capacity of educators in EmONC skills and information and communication technology for effective teaching and learning is mandatory. Related challenges with individual technical capacity, technological deficiencies and infrastructure to support the technological advancement have been reported in other studies that have used a blended learning approach [ 58 ]. Resource constraints – financial and infrastructural (e.g. computers) as well as internet access are key challenges to participation in CPD activities especially the self-directed learning [ 16 ]. Designing self-directed modules that can be accessed and completed offline will increase access especially in poorly connected settings with electric power and network coverage.

Strengths and limitations

This study assessed the feasibility a blended midwifery educator CPD programme in low resource settings. This was conducted in a multi-country and multi-site context which provided opportunities for learning across the two countries, two levels of training institutions and specific in-country experiences [ 20 ]. The study served to improve awareness of the availability of the CPD programme so that (1) regulators can ensure that midwifery educators take this as part of mandatory CPD required for relicensing and (2) training institutions can plan to support their educators access/participate in the practical components of the programme after the study. It is a mandatory requirement of the Nursing and Midwifery Councils of Kenya and Nigeria for nurse/midwives and midwifery educators to demonstrate evidence of CPD for renewal of practising license [ 40 , 41 ]. The use of mixed methods research design with multiple evaluations was relevant to address the aims and objectives of the study and ensure methodological rigour, depth and scientific validity as recommended for good practice in designing pilot studies [ 37 , 38 ]. This also enhanced triangulation of findings and enabled the capturing of broad perspectives important in strengthening sustainable implementation of the blended CPD programme [ 39 ]. Preliminary findings were disseminated to participant stakeholders from Kenya and Nigeria at the knowledge management and learning event in Nairobi. This approach enhanced the credibility and trustworthiness of the final findings reported. We believe our study findings from different participants using multiple data collection methods are robust, transparent and trustworthy for generalization to other contexts [ 38 ].The self-directed learning component of the blended CPD programme is hosted on the WCEA platform which is accessible to healthcare professionals in over 60 countries in Africa, Asia and Middle East and accredited for continuous professional development (59). Although our sample size was small, it is sufficient, geographically representative for training institutions across the countries and acceptable for feasibility studies [ 34 ].

The additional cost analysis of implementing the blended midwifery educator CPD programme is relevant and key to the uptake, scale-up and sustainability of the programme but this was not conducted due to limited funding. Different CPD programme funding models exist. In Nigeria, educators are required to meet the costs for accessing and completing the CPD programme components, while in Kenya the cost of accessing the online component is minimal (internet access costs only) and the face-to-face component has to be funded. The cost of implementing the programme should be explored in future studies and optional models for sustainable funding explored with stakeholders.

Implications

Our findings show demand for the CPD programme. Regular continuous professional development could help to bridge the gap between theory and practice and improve the quality of teaching by midwifery educators. A blended CPD programme is effective in improving the teaching and clinical skills of midwifery educators and increasing their confidence in effective teaching. However, midwifery educators require motivation and close support (individual capacity, time, technological infrastructure and policy) if the blended CPD approach is to be mandatory and successfully implemented in resource limited settings. Besides, regular quality assurance modalities including review of content, monitoring and evaluation of uptake of the CPD programme should be undertaken to ensure that updated and relevant content is available.

For quality CPD programmes, hands-on teaching is more effective than didactic classroom teaching and should be used when feasible to transfer clinical skills. Distance education models (self-directed learning) in combination with short residential training and mentoring should be embraced to strengthen capacity strengthening of midwifery educators; and CPD programmes must consider the local context in which participants work and teach [ 16 , 23 ]. Evidence has shown that knowledge and clinical skills are retained for up to 12 months after training [ 54 ]. Taking the CPD programme annually will potentially maintain/improve knowledge, skills and practice by midwifery educators for quality teaching and learning leading to a competent midwifery workforce.

For quality midwifery education and practice, educators need contact with clinical practice to strengthen classroom teaching [ 6 , 7 ]. This will promote and enable students to acquire the skills, knowledge, and behaviours essential to become autonomous midwifery practitioners. Therefore, demonstrating relevant practical clinical CPD should be included in midwifery educator CPD policy. In addition, a business case by the CPD hubs on the sustainability of the face-to-face practical components in the centres is necessary. Stakeholder engagement on cost and sustainability are required as key policy components for the scale-up of the blended midwifery educator CPD programme for impact.

The blended CPD programme was relevant, acceptable and feasible to implement. Midwifery educators reacted positively to its content as they were very satisfied with the modules meeting their needs and rated the content as relevant to their practice. The programme also improved their knowledge, confidence and skills in teaching, students’ assessments and providing effective feedback for learning and using digital/technological innovations for effective teaching and learning. Investments in information and communication technology, quality assurance and academic integrity were highlighted as critical components for the scale-up of the programme. For successful and mandatory implementation of the specific midwifery educator CPD programme to enhance practice, a policy framework by midwifery training regulators is required by countries.

Data availability

The datasets generated and/or analysed during the current study are not publicly available due to the confidentiality of the data but are available from the corresponding author on request.

Renfrew MJ, McFadden A, Bastos MH, Campbell J, Channon AA, Cheung NF, et al. Midwifery and quality care: findings from a new evidence-informed framework for maternal and newborn care. Lancet. 2014;384(9948):1129–45.

Article   Google Scholar  

World Health Organization, United Nations Population Fund, International Confederation of Midwives. The State of the World’s Midwifery 2021: Building a health workforce to meet the needs of women, newborns and adolescents everywhere 2021. https://www.unfpa.org/publications/sowmy-2021 .

Filby A, McConville F, Portela A. What prevents quality midwifery care? A systematic mapping of barriers in low and middle income countries from the provider perspective. PLoS ONE. 2016;11(5):e0153391.

WHO. Strengthening quality midwifery education for Universal Health Coverage 2030: Framework for action 2019. https://www.who.int/publications/i/item/9789241515849 .

United Nations Population Fund, International Confederation of Midwives, World Health Organization. The State of the World’s Midwifery 2021: Building a health workforce to meet the needs of women, newborns and adolescents everywhere 2021. https://www.unfpa.org/publications/sowmy-2021 .

International Confederation of Midwives. ICM Global Standards for Midwifery Education. (Revised 2021) 2021. https://www.internationalmidwives.org/assets/files/education-files/2021/10/global-standards-for-midwifery-education_2021_en-1.pdf .

WHO. Midwifery educator core competencies: building capacities of midwifery educators 2014. https://www.who.int/hrh/nursing_midwifery/14116_Midwifery_educator_web.pdf .

Gavine A, MacGillivray S, McConville F, Gandhi M, Renfrew MJ. Pre-service and in-service education and training for maternal and newborn care providers in low-and middle-income countries: an evidence review and gap analysis. Midwifery. 2019;78:104–13.

Shikuku DN, Tallam E, Wako I, Mualuko A, Waweru L, Nyaga L, et al. Evaluation of capacity to deliver Emergency Obstetrics and Newborn Care updated midwifery and reproductive health training curricula in Kenya: before and after study. Int J Afr Nurs Sci. 2022;17:100439.

Google Scholar  

International Confederation of Midwives. Global Standards for Midwifery Regulation. 2011. https://www.internationalmidwives.org/assets/files/regulation-files/2018/04/global-standards-for-midwifery-regulation-eng.pdf .

World Health Organization. Global strategic directions for nursing and midwifery 2021–2025. Geneva: World Health Organization. 2021. https://iris.who.int/bitstream/handle/10665/344562/9789240033863-eng.pdf?sequence=1 .

Smith RM, Gray JE, Homer CSE. Common content, delivery modes and outcome measures for faculty development programs in nursing and midwifery: a scoping review. Nurse Educ Pract. 2023:103648.

Nursing and Midwifery Board of Australia. Registration standard: Continuing professional development 2016 3rd January 2022. https://www.nursingmidwiferyboard.gov.au/Registration-Standards/Continuing-professional-development.aspx .

International Confederation of Midwives. Essential competencies for midwifery practice: 2019 Update. 2019.

Baloyi OB, Jarvis MA. Continuing Professional Development status in the World Health Organisation, Afro-region member states. Int J Afr Nurs Sci. 2020;13:100258.

Mack HG, Golnik KC, Murray N, Filipe HP. Models for implementing continuing professional development programs in low-resource countries. MedEdPublish. 2017;6(1).

Lucas A. Continuous professional development, friend or foe? Br J Midwifery. 2012;20(8):576–81.

Ingwu JA, Efekalam J, Nwaneri A, Ohaeri B, Israel C, Chikeme P, et al. Perception towards mandatory continuing professional development programme among nurses working at University of Nigeria Teaching Hospital, Enugu-Nigeria. Int J Afr Nurs Sci. 2019;11:100169.

Hasumi T, Jacobsen KH. Healthcare service problems reported in a national survey of South africans. Int J Qual Health Care. 2014;26(4):482–9.

Giri K, Frankel N, Tulenko K, Puckett A, Bailey R, Ross H. Keeping up to date: continuing professional development for health workers in developing countries. IntraHealth Int. 2012.

Botha A, Booi V, editors. mHealth implementation in South Africa. 2016 IST-Africa Week Conference; 2016: IEEE.

World Continuing Education Alliance (WCEA). World Continuing Education Alliance: About us2022 3rd January 2022. https://lmic.wcea.education/about-us/ .

West F, Homer C, Dawson A. Building midwifery educator capacity in teaching in low and lower-middle income countries. A review of the literature. Midwifery. 2016;33:12–23.

van Wyk JM, Wolvaardt JE, Nyoni CN. Evaluating the outcomes of a faculty capacity development programme on nurse educators in sub-saharan Africa. Afr J Health Professions Educ. 2020;12(4):201–5.

Frantz JM, Bezuidenhout J, Burch VC, Mthembu S, Rowe M, Tan C, et al. The impact of a faculty development programme for health professions educators in sub-saharan Africa: an archival study. BMC Med Educ. 2015;15(1):1–8.

Fullerton JT, Johnson PG, Thompson JB, Vivio D. Quality considerations in midwifery pre-service education: exemplars from Africa. Midwifery. 2011;27(3):308–15.

Shikuku DN, Tallam, E., Wako, I., Mualuko, A., Waweru, L., Nyaga, L., ... & Ameh, C. Evaluation of capacity to deliver Emergency Obstetrics and Newborn Care updated midwifery and reproductive health training curricula in Kenya: Before and after study. 2022

Shikuku DN, Jebet J, Nandikove P, Tallam E, Ogoti E, Nyaga L, et al. Improving midwifery educators’ capacity to teach emergency obstetrics and newborn care in Kenya universities: a pre-post study. BMC Med Educ. 2022;22(1):1–10.

Akiode A, Fetters T, Daroda R, Okeke B, Oji E. An evaluation of a national intervention to improve the postabortion care content of midwifery education in Nigeria. Int J Gynecol Obstet. 2010;110(2):186–90.

Nursing Council of Kenya. Continuing Professional Development guidelines. Nursing Council of Kenya; 2021.

Nursing and Midwifery Council of Nigeria. Promoting & Maintaining Excellence in Nursing Education and Practice: Functions2022. https://www.nmcn.gov.ng/function.html .

Ministry of Health. Kenya Health Policy 2014–2030: Towards attaining the highest standard of health 2014. http://publications.universalhealth2030.org/uploads/kenya_health_policy_2014_to_2030.pdf .

Orsmond GI, Cohn ES. The distinctive features of a feasibility study: objectives and guiding questions. OTJR: Occupation Participation Health. 2015;35(3):169–77.

Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. How we design feasibility studies. Am J Prev Med. 2009;36(5):452–7.

Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol. 2010;10(1):1–7.

Kirkpatrick DL. Implementing the four levels: A practical guide for effective evaluation of training programs: Easyread super large 24pt edition: ReadHowYouWant. com; 2009.

Warfa A-RM. Mixed-methods design in biology education research: Approach and uses. CBE—Life Sci Educ. 2016;15(4):rm5.

Creswell JW, Creswell JD. Research design: qualitative, quantitative, and mixed methods approaches. Sage; 2017.

Creswell JW, Clark VLP. Designing and conducting mixed methods research. Third ed: Sage; 2018.

NCK Online CPD Portal: Continuous Professional Development [Internet]. 2021. https://osp.nckenya.com/cpd? .

Nursing and Midwifery Council of Nigeria. Promoting & maintaining Excellence in nursing education and practice: Renewal of License. 2022. Available from. https://www.nmcn.gov.ng/renewal.html .

Warren N, Gresh A, Mkhonta NR, Kazembe A, Engelbrecht S, Feraud J et al. Pre-service midwifery education in sub-saharan Africa: a scoping review. Nurse Educ Pract. 2023:103678.

Malterud K, Siersma VD, Guassora AD. Sample size in qualitative interview studies: guided by information power. Qual Health Res. 2016;26(13):1753–60.

Muellmann S, Brand T, Jürgens D, Gansefort D, Zeeb H. How many key informants are enough? Analysing the validity of the community readiness assessment. BMC Res Notes. 2021;14(1):1–6.

Hennink M, Kaiser BN. Sample sizes for saturation in qualitative research: a systematic review of empirical tests. Soc Sci Med. 2022;292:114523.

Shumway JM, Harden RM. AMEE Guide 25: the assessment of learning outcomes for the competent and reflective physician. Med Teach. 2003;25(6):569–84.

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3(2):77.

Erlandsson K, Doraiswamy S, Wallin L, Bogren M. Capacity building of midwifery faculty to implement a 3-years midwifery diploma curriculum in Bangladesh: A process evaluation of a mentorship programme. Nurse Educ Pract. 2018;29:212–8.

Erlandsson K, Byrskog U, Osman F, Pedersen C, Hatakka M, Klingberg-Allvin M. Evaluating a model for the capacity building of midwifery educators in Bangladesh through a blended, web-based master’s programme. Global Health Action. 2019;12(1):1652022.

Hatakka M, Osman F, Erlandsson K, Byrskog U, Egal J, Klingberg-Allvin M. Change-makers in midwifery care: exploring the differences between expectations and outcomes—A qualitative study of a midwifery net-based education programme in the Somali region. Midwifery. 2019;69:135–42.

Erlandsson K, Osman F, Hatakka M, Egal JA, Byrskog U, Pedersen C, et al. Evaluation of an online master’s programme in Somaliland. A phenomenographic study on the experience of professional and personal development among midwifery faculty. Nurse Educ Pract. 2017;25:96–103.

Bogren M, Rosengren J, Erlandsson K, Berg M. Build professional competence and equip with strategies to empower midwifery students–an interview study evaluating a simulation-based learning course for midwifery educators in Bangladesh. Nurse Educ Pract. 2019;35:27–31.

Msosa A, Msiska M, Mapulanga P, Mtambo J, Mwalabu G. Simulation-based education in classroom and clinical settings in sub-saharan Africa: a systematic review. Higher Education, Skills and Work-Based Learning; 2023.

Ameh CA, White S, Dickinson F, Mdegela M, Madaj B, van den Broek N. Retention of knowledge and skills after emergency Obstetric Care training: a multi-country longitudinal study. PLoS ONE. 2018;13(10):e0203606.

Evans C, Razia R, Cook E. Building nurse education capacity in India: insights from a faculty development programme in Andhra Pradesh. BMC Nurs. 2013;12(1):1–8.

Koto-Shimada K, Yanagisawa S, Boonyanurak P, Fujita N. Building the capacity of nursing professionals in Cambodia: insights from a bridging programme for faculty development. Int J Nurs Pract. 2016;22:22–30.

Kitema GF, Laidlaw A, O’Carroll V, Sagahutu JB, Blaikie A. The status and outcomes of interprofessional health education in sub-saharan Africa: a systematic review. J Interprof Care. 2023:1–23.

Ladur AN, Kumah EA, Egere U, Mgawadere F, Murray C, Ravit M et al. A blended learning approach for capacity strengthening to improve the quality of integrated HIV, TB, and Malaria services during antenatal and postnatal care in LMICs: a feasibility study. medRxiv. 2023:2023.05. 04.23289508.

World Continuing Education Alliance (WCEA). Improving Health Outcomes: WCEA delivering sustainable solutions for CPD & lifelong learning2023 26th December 2023. https://wcea.education/ .

Download references

Acknowledgements

The study was made possible through the financial support of the Johnson and Johnson Foundation for the three-year “Design, implementation and evaluation of Nursing/Midwifery CPD Educator Programme in Kenya” (2021 – 2023) and the Alliance to Improve Midwifery Education through UNFPA Headquarters. Special acknowledgement to nursing and midwifery educators from mid-level training colleges and universities in Kenya and Nigeria, Ministries of Health, Nursing Council of Kenya, Nursing and Midwifery Council of Nigeria, KMTC headquarters management who participated in the study. Also, we specially appreciate the World Continuing Education Alliance for the dedicated support with the online modules and expert trainers who participated in the delivery of the face-to-face training component: Aisha Hassan, Dr. Mojisola Ojibara, Dr. Eniola Risikat Kadir, Aminat Titi Kadir, Benson Milimo, Esther Ounza, Marthar Opisa, Millicent Kabiru, Sylvia Kimutai, Dr. Joyce Jebet, Dr. Steve Karangau, Dr. Moses Lagat and Dr. Evans Ogoti. Gratitude to Boslam Adacha and Roselynne Githinji for their dedicated support with data preparation for analysis and Dr. Sarah White for her statistical analysis expert guidance and support. Thank you also to Geeta Lal at UNFPA Headquarters. Lastly, the authors would like to acknowledge the special technical and logistical support provided by the LSTM – Kenya team (Onesmus Maina, Martin Eyinda, David Ndakalu, Diana Bitta, Esther Wekesa and Evans Koitaba) and LSTM Nigeria team (Dr. Michael Adeyemi and Deborah Charles) during the trainings.

The study was funded by the Johnson and Johnson Foundation as part of the three-year “Design, implementation and evaluation of Nursing/Midwifery CPD Educator Programme in Kenya” and the Alliance to Improve Midwifery Education through UNFPA. The Johnson and Johnson Foundation were not involved in the research – study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.

Author information

Authors and affiliations.

Liverpool School of Tropical Medicine (Kenya), P.O. Box 24672-00100, Nairobi, Kenya

Duncan N. Shikuku & Lucy Nyaga

Liverpool School of Tropical Medicine (UK), Liverpool, L3 5QA, UK

Duncan N. Shikuku, Alice Norah Ladur, Carol Bedwell & Charles Ameh

Liverpool School of Tropical Medicine (Nigeria), Utako District, P.O Box 7745, Abuja, Nigeria

Hauwa Mohammed

Moi University, P.O. Box 4606-30100, Eldoret, Kenya

Lydia Mwanzia

Masinde Muliro University of Science and Technology, P.O. Box 190-50100, Kakamega, Kenya

Peter Nandikove

Maseno University, P.O. Box 3275-40100, Kisumu, Kenya

Alphonce Uyara

Kenya Medical Training College, P.O Box 30195-00100, Nairobi, Kenya

Catherine Waigwe

Department of Family Health, Ministry of Health (Kenya), P.O. Box 30016-00100, Nairobi, Kenya

Issak Bashir

Aga Khan University of East Africa, P.O Box 39340-00623, Nairobi, Kenya

Eunice Ndirangu

Burnet Institute, 85 Commercial Road Prahran Victoria, Melbourne, Australia

Sarah Bar-Zeev

University of Nairobi, P. O. Box 19676-00100, Nairobi, Kenya

Charles Ameh

Diponegoro University, JI. Prof Sudarto No 13, Temalang, Kec, Tembalang, Kota, Semarang, Jawa Tengah, 50275, Indonesia

You can also search for this author in PubMed   Google Scholar

Contributions

DNS, SBZ and CA conceived the idea and designed the study protocol; DNS designed the online data collection tools/checklists/assessments, performed data extraction, cleaning, analysis and interpretation of the results, drafted the primary manuscript, reviewed and prepared it for publication; DNS, HM, LM, PN and AU conducted the training intervention, collected data and reviewed the drafts and final manuscript; AL participated in the design of the study, qualitative data analysis, interpretation of findings and reviewed draft manuscripts; CW, LN, IB, EN, CB and SBZ participated in the design of the study procedures and substantively reviewed the drafts and final manuscript. CA reviewed study procedures, data collection tools, provided oversight in investigation, analysis, interpretation and substantively reviewed the manuscript drafts. SBZ and CA obtained funding for the study. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Duncan N. Shikuku .

Ethics declarations

Ethics approval and consent to participate.

Ethics review and approvals were obtained from Liverpool School of Tropical Medicine’s Research Ethics Committee (LSTM REC No. 23 − 004) and in-country ethical approvals from Kenya (MTRH/MU – IREC FAN 0004383; NACOSTI License No: NACOSTI/P/23/25498) and Nigeria (NHREC Approval Number NHREC/01/01/2007- 31/03/2023). Participation in the study was strictly voluntary and did not form part of the educator’s performance appraisals. Not taking part in the study did not disadvantage some educators who consented but missed out. Informed electronic and written consent was obtained from all participants. Unique participant codes were used for identification and all the data collection tools/forms and datasets were de-identified with no participant identifying information. All interviews were conducted at the offices of the respective stakeholders maintaining privacy during data collection process.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Shikuku, D.N., Mohammed, H., Mwanzia, L. et al. Evaluation of the feasibility of a midwifery educator continuous professional development (CPD) programme in Kenya and Nigeria: a mixed methods study. BMC Med Educ 24 , 534 (2024). https://doi.org/10.1186/s12909-024-05524-w

Download citation

Received : 24 January 2024

Accepted : 06 May 2024

Published : 14 May 2024

DOI : https://doi.org/10.1186/s12909-024-05524-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Continuous professional development
• Feasibility

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]