needs assessment research article

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

A guiding framework for needs assessment evaluations to embed digital platforms in partnership with Indigenous communities

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Supervision, Writing – original draft

Affiliation School of Occupational and Public Health, Toronto Metropolitan University, Toronto, ON, Canada

Roles Data curation, Formal analysis, Investigation, Software, Visualization, Writing – original draft

Affiliation School of Public Health Sciences, University of Waterloo, Waterloo, ON, Canada

Roles Conceptualization, Investigation, Project administration, Resources, Supervision, Writing – review & editing

Affiliation Île-à-la-Crosse School Division, The Northern Village of Île-à-la-Crosse, Île-à-la-Crosse, SK, Canada

Roles Conceptualization, Investigation, Resources, Supervision

Roles Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – review & editing

* E-mail: [email protected]

Affiliations DEPtH Lab, Faculty of Health Sciences, Western University, London, ON, Canada, Department of Epidemiology and Biostatistics, Schulich School of Medicine and Dentistry, Western University, London, ON, Canada, Lawson Health Research Institute, London, Ontario, Canada

• Jasmin Bhawra, 
• M. Claire Buchan, 
• Brenda Green, 
• Kelly Skinner, 
• Tarun Reddy Katapally

• Published: December 22, 2022
• https://doi.org/10.1371/journal.pone.0279282
• Reader Comments

Introduction

In community-based research projects, needs assessments are one of the first steps to identify community priorities. Access-related issues often pose significant barriers to participation in research and evaluation for rural and remote communities, particularly Indigenous communities, which also have a complex relationship with academia due to a history of exploitation. To bridge this gap, work with Indigenous communities requires consistent and meaningful engagement. The prominence of digital devices (i.e., smartphones) offers an unparalleled opportunity for ethical and equitable engagement between researchers and communities across jurisdictions, particularly in remote communities.

This paper presents a framework to guide needs assessments which embed digital platforms in partnership with Indigenous communities. Guided by this framework, a qualitative needs assessment was conducted with a subarctic Métis community in Saskatchewan, Canada. This project is governed by an Advisory Council comprised of Knowledge Keepers, Elders, and youth in the community. An environmental scan of relevant programs, three key informant interviews, and two focus groups (n = 4 in each) were conducted to systematically identify community priorities.

Through discussions with the community, four priorities were identified: (1) the Coronavirus pandemic, (2) climate change impacts on the environment, (3) mental health and wellbeing, and (4) food security and sovereignty. Given the timing of the needs assessment, the community identified the Coronavirus pandemic as a key priority requiring digital initiatives.

Recommendations for community-based needs assessments to conceptualize and implement digital infrastructure are put forward, with an emphasis on self-governance and data sovereignty.

Citation: Bhawra J, Buchan MC, Green B, Skinner K, Katapally TR (2022) A guiding framework for needs assessment evaluations to embed digital platforms in partnership with Indigenous communities. PLoS ONE 17(12): e0279282. https://doi.org/10.1371/journal.pone.0279282

Editor: Stephane Shepherd, Swinburne University of Technology, AUSTRALIA

Received: June 1, 2022; Accepted: December 2, 2022; Published: December 22, 2022

Copyright: © 2022 Bhawra et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data are co-owned by the community and all data requests should be approved by the Citizen Scientist Advisory Council and the University of Regina Research Office. Citizen Scientist Advisory Council Contact: Mr. Duane Favel, Mayor of Ile-a-lacrosse, email: [email protected] ; [email protected] University of Regina Research Office contact: Ara Steininger, Research Compliance Officer; E-mail: [email protected] . Those interested can access the data in the same manner as the authors.

Funding: TRK received funding from the Canadian Institutes of Health Research (CIHR) and the Canada Research Chairs Program to conduct this research. The funding organization had no role to play in any part of the study implementation of manuscript generation.

Competing interests: The authors have declared that no competing interests exist.

Community engagement has been the cornerstone of participatory action research in a range of disciplines. Every community has a unique culture and identity, hence community members are the experts regarding their diverse histories, priorities, and growth [ 1 – 3 ]. As a result, the successful uptake, implementation, and longevity of community-based research initiatives largely depends on meaningful community engagement [ 4 – 9 ]. There is a considerable body of evidence establishing the need for ethical community-research partnerships which empower citizens and ensure relevant and sustainable solutions [ 1 – 3 , 10 ]. For groups that have been marginalized or disadvantaged, community-engaged research that prioritizes citizens’ control in the research process can provide a platform to amplify citizens’ voices and ensure necessary representation in decision-making [ 11 ]. Such initiatives must be developed in alignment with a community’s cultural framework, expectations, and vision [ 12 ] to support continuous and meaningful engagement throughout the project. In particular, when partnering with Indigenous communities, a Two-Eyed Seeing approach can provide valuable perspective to combine the strengths of Indigenous and Western Knowledges, including culturally relevant methods, technologies, and tools [ 13 – 15 ].

Many communities have a complicated relationship with research as a result of colonialism, and the trauma of exploitation and discrimination has continued to limit the participation of some communities in academic partnerships [ 16 ]. Indigenous Peoples in Canada experience a disproportionate number of health, economic, and social inequalities compared to non-Indigenous Canadians [ 17 ]. Many of these health (e.g., elevated risk of chronic and communicable diseases) [ 18 – 21 ]), socioeconomic (e.g., elevated levels of unemployment and poverty) [ 19 , 22 – 24 ], and social (e.g., racism and discrimination) [ 19 , 22 – 24 ]) inequities can be traced back to the long-term impacts of assimilation, colonization, residential schools, and a lack of access to healthcare [ 19 , 20 , 22 – 24 ]. To bridge this gap, and more importantly, to work towards Truth and Reconciliation [ 25 ], work with Indigenous Peoples must be community-driven, and community-academia relationship building is essential before exploring co-conceptualization of initiatives [ 26 ].

One of the first steps in building a relationship is to learn more about community priorities by conducting a needs assessment [ 27 , 28 ]. A needs assessment is a research and evaluation method for identifying areas for improvement or gaps in current policies, programs, and services [ 29 ]. When conducted in partnership with a specific community, needs assessments can identify priorities and be used to develop innovative solutions, while leveraging the existing knowledge and systems that communities have in place [ 30 ]. Needs assessments pave the path for understanding the value and applicability of research for community members, incorporating key perspectives, and building authentic partnerships with communities to support effective translation of research into practice.

For rural, remote, and northern communities within Canada, issues related to access (e.g., geographic location, transportation, methods of communication, etc.) pose significant barriers to participation in research and related initiatives [ 31 ]. Digital devices, and in particular, the extensive usage of smartphones [ 32 ] offers a new opportunity to ethically and equitably engage citizens [ 33 ]. Digital platforms (also referred to as digital tools) are applications and software programs accessible through digital devices. Digital platforms can be used for a variety of purposes, ranging from project management, to healthcare delivery or mass communication [ 34 ]. Digital infrastructure–the larger systems which support access and use of these digital platforms, including internet, satellites, cellular networks, and data storage centres [ 34 ]. The Coronavirus (COVID-19) pandemic has catalyzed the expansion of digital technology, infrastructure and the use of digital devices in delivering essential services (e.g., healthcare) and programs to communities [ 35 , 36 ].

While digital platforms have been used in Indigenous communities for numerous initiatives, including environmental mapping initiatives (e.g., research and monitoring, land use planning, and wildlife and harvest studies) [ 37 , 38 ] and telehealth [ 39 ], there has largely been isolated app development without a corresponding investment in digital infrastructure. This approach limits the sustainability of digital initiatives, and importantly does not acknowledge an Indigenous world view of holistic solutions [ 39 ].

Thus given the increasing prominence of digital devices [ 39 , 40 ], it is critical to evaluate the conceptualization, implementation, and knowledge dissemination of digital platforms. To date, there is little guidance on how to evaluate digital platforms, particularly in partnership with rural and remote communities [ 41 ]. A review of recent literature on community-based needs assessments uncovered numerous resources for conducting evaluations of digital platforms, however, a key gap is the lack of practical guidance for conducting needs assessments in close collaboration with communities in ways that acknowledge existing needs, resources, supports and infrastructure that also incorporates the potential role of digital platforms in addressing community priorities.

This paper aims to provide researchers and evaluators with a framework (step-by-step guide) to conduct needs assessments for digital platforms in collaboration with Indigenous communities. To achieve this goal, a novel needs assessment framework was developed using a Two-Eyed Seeing approach [ 13 – 15 ] to enable the identification of community priorities, barriers and supports, as well as existing digital infrastructure to successfully implement digital solutions. To demonstrate the application of this framework, a community-engaged needs assessment conducted with a subarctic Indigenous community in Canada is described and discussed in detail.

Framework design and development

This project commenced with the design and development of a new framework to guide community-based needs assessments in the digital age.

Needs assessments

Needs assessments are a type of formative evaluation and are often considered a form of strategic or program planning, even more than they are considered a type of evaluation. Needs assessments can occur both before and during an evaluation or program implementation; however, needs assessments are most effective when they are conducted before a new initiative begins or before a decision is made about what to do (e.g., how to make program changes) [ 29 ]. Typically, a needs assessment includes: 1) collecting information about a community; 2) determining what needs are already being met; and 3) determining what needs are not being met and what resources are available to meet those needs [ 42 ].

Framework development

Based on existing literature, community consultation, and drawing expertise from our team of evaluation experts who have over a decade of experience working with Indigenous communities on a range of research and evaluation projects, a novel framework was developed to guide community-based needs assessments focused on the application of digital platforms.

This framework (see Fig 1 ) is driven by core questions necessary to identify community priorities that can be addressed by developing and implementing digital platforms. Through team discussion and community consultation, five key topic areas for the assessment of community needs were identified: i) current supports; ii) desired supports; iii) barriers; iv) community engagement; and v) digital access and connectivity. A series of general questions across the five needs assessment topic areas were developed. Thereafter, a set of sub-questions were embedded in each key topic area.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0279282.g001

The Guiding Framework outlines an approach for conducting community needs assessments which can be adapted across communities and jurisdictions. This framework offers a flexible template that can be used iteratively and applied to various community-engaged needs assessments in a range of areas, including but not limited to community health and wellness projects. The questions assigned to each topic area can be used to guide needs assessments of any priority identified by community stakeholders as suitable for addressing with digital platforms.

Needs assessment methods

The Guiding Framework was implemented in collaboration with a subarctic Indigenous community in Canada, and was used to identify key community priorities, barriers, supports, and existing digital infrastructure which could inform the design and implementation of tailored digital platforms.

Using an environmental scan of relevant documents and qualitative focus groups and interviews, a needs assessment was conducted with the Northern Village of Île-à-la-Crosse, Saskatchewan, Canada between February and May 2020.

This project is governed by a Citizen Scientist Advisory Council which included researchers, Knowledge Keepers, Elders, and youth from Île-à-la-Crosse. The study PI (TRK) and Co-Investigator (JB) developed a relationship with key decision-makers in Île-à-la-Crosse in 2020. Through their guidance and several community visits, the decision-makers introduced the research team to Elders, youth, and other community members to gain a better understanding of current priorities and needs in Île-à-la-Crosse. The research team developed relationships with these community members and invited them to join the Council to formally capture feedback and plan ongoing projects to promote health and wellbeing in the community. The Council represents the needs and interests of the community, and guides the project development, implementation, and evaluation. Council members were provided with Can $150 (US $119.30) as honoraria for each meeting to respect their time, knowledge, and contributions.

Written consent was obtained from all focus group participants and verbal consent was obtained from all key informants participating in interviews. This study received ethics clearance from the research ethics boards of the University of Regina and the University of Saskatchewan through a synchronized review protocol (REB# 2017–29).

Established in 1776, Île-à-la-Crosse is a northern subarctic community with road access in northwest Saskatchewan. Sakitawak, the Cree name for Île-à-la-Crosse, means “where the rivers meet,” hence the community was an historically important meeting point for the fur trade in the 1800s [ 43 , 44 ] The community lies on a peninsula on the Churchill River, near the intersections with the Beaver River and Canoe River systems. Île-à-la-Crosse has a rich history dating back to the fur trade. Due to its strategic location, Montreal-based fur traders established the first trading point in Île-à-la-Crosse in 1776, making the community Saskatchewan’s oldest continually inhabited community next to Cumberland House [ 45 ]. In 1821, Île-à-la-Crosse became the headquarters for the Hudson’s Bay Company’s operations in the territory. In 1860, the first convent was established bringing Western culture, medical services, and education to the community.

Île-à-la-Crosse has a population of roughly 1,300 people [ 19 ]. Consistent with Indigenous populations across Canada, the average age of the community is 32.7 years, roughly 10 years younger than the Canadian non-Indigenous average [ 19 ]. Census data report that just under half (44%) of the community’s population is under the age of 25, 46.3% are aged 25–64, and 9.3% aged 65 and over [ 19 ]. Members of the community predominantly identify as Métis (77%), with some identifying as First Nations (18%), multiple Indigenous responses (1.2%), and non-Indigenous (2.7%) [ 19 ]. Many community members are employed in a traditional manner utilizing resources of the land (e.g., hunting, fishing, trapping), others in a less traditional manner (e.g., lumbering, tourism, wild rice harvesting), and some are employed through the hospital and schools. The community currently has one elementary school with approximately 200 students from preschool to Grade 6, and one high school serving Grades 7–12 with adult educational programming. Île-à-la-Crosse has a regional hospital with Emergency Services, which includes a health services centre with a total of 29 beds. Other infrastructure of the community includes a Royal Canadian Mounted Police (RCMP) station, a village office, volunteer fire brigade, and a catholic church [ 46 ].

Needs assessment approach

Île-à-la-Crosse shared their vision of integrating digital technology and infrastructure as part of its growth, thus the needs assessment was identified as an appropriate method to provide the formative information necessary to understand what the needs are, including who (i.e., players, partners), and what (i.e., information sources) would need to be involved, what opportunities exist to address the needs, and setting priorities for action with key community stakeholders [ 47 ]. As a starting point and rationale for this needs assessment, the community of Île-à-la-Crosse values the potential of technology for improving health communication, information reach, access to resources, and care, and was interested in identifying priorities to begin building digital infrastructure. Given the timing of the COVID-19 pandemic, being responsive to community health needs were key priorities that they wanted to start addressing using a digital platform. This needs assessment facilitated and enabled new conversations around key priorities and next steps.

The evaluation approach was culturally-responsive and included empowerment principles [ 48 – 50 ]. Empowerment evaluation intends to foster self-determination. The empowerment approach [ 50 ] involved community members–represented through the Citizen Scientist Advisory Council–engaging in co-production of the evaluation design and implementation by establishing key objectives for the evaluation, informing evaluation questions, building relevant and culturally responsive indicators, developing focus group guides, leading recruitment and data collection, and interpreting results [ 51 ]. In this way, the approach incorporated local community and Indigenous Knowledges as well as Western knowledge, in a similar approach to Two-Eyed Seeing [ 13 – 15 ]. Using these needs assessment evaluation results, the community will identify emerging needs and potential application issues, and work with the researchers to continue shaping project development and implementation.

Two-Eyed Seeing to embed digital platforms

Two-Eyed Seeing as described by Elder Albert Marshall [ 13 , 14 ], refers to learning to see with the strengths of Indigenous and Western Knowledges. Our engagement and overall approach to working with the community of Île-à-la-Crosse takes a Two-Eyed Seeing lens, from co-conceptualization of solutions, which starts with understanding the needs of the community. All needs are a result of direct Indigenous Knowledge that was provided by the Advisory Council. Indigenous Knowledge is not limited to the knowledge of Elders and Traditional Knowledge Keepers; however, they play a critical role in guiding that knowledge through by providing historical, geographic, and cultural context. Moreover, the Knowledge Keepers can be key decision-makers in the community, and in our case, they were key informants who participated in this needs assessment. Every aspect of needs assessment was dependent on the Advisory Council and Key informants providing the Indigenous Knowledge that the research team needed to tailor digital solutions. As a result, Two-Eyed Seeing approach informed all aspects of the research process.

As we are working to develop, and bring digital platforms and technologies (i.e., Western methods) to address key community priorities, Indigenous Knowledge is central to the overall project. Indigenous Elders, decision-makers, and Advisory Council members are bringing both their historical and lived experience to inform project goals, key priority areas, target groups, and methods. Île-à-la-Crosse is a predominantly Metis community, which differs in culture from other Indigenous communities in Canada—First Nations and Inuit communities. Ceremony is not a key part of community functioning; thus, specific cultural ceremonies were not conducted upon advice of the Advisory Council. Instead, the knowledge of historical issues, challenges, and success stories in the community is considered Indigenous Knowledge for this needs assessment, and more importantly, this Indigenous Knowledge informed the focus areas and next steps for this project. Overall, the spirit of collaboration and co-creation which combined Western research methods/technology with Indigenous Knowledge and expertise is considered Two-Eyed Seeing in this project. This lens was taken at all phases, from the engagement stage to Advisory Council meetings, to planning and executing the needs assessment and next steps.

Data collection

In order to obtain an in-depth understanding of the key priorities and supports within the community of Île-à-la-Crosse, this needs assessment used a qualitative approach. An environmental scan was conducted in February 2020 of current school and community policies and programs. Published reports, meeting memos, community social media accounts, and the Île-à-la-Crosse website were reviewed for existing policies and programs. The Citizen Scientist Advisory Council identified appropriate data sources for the document review and corroborated which programs and initiatives were currently active in the community.

Qualitative data were collected from key decision-makers and other members within the community. A purposeful convenience sampling approach was employed to identify members of the community who could serve on the Council and participate in focus group discussions. Key decision makers and existing Council members recommended other community members who could join the focus group discussions to provide detailed and relevant information on community priorities, digital infrastructure, supports, and challenges. Two focus groups were conducted by members of the research team in Île-à-la-Crosse with the Council in May 2020. Focus group participants were asked to describe community priorities, supports, and barriers, as well as experience and comfort with digital platforms. Each focus group had four participants, were two-hours in length, and followed an unstructured approach. Three key informant interviews were conducted in Île-à-la-Crosse between February and April 2020. One-hour interviews were conducted one-on-one and followed a semi-structured interview format. The focus groups and key informant interviews were led by the study PI, TRK, and Co-Investigator, JB, who have extensive training and experience with qualitative research methods, particularly in partnership with Indigenous communities. Focus groups and key informant interviews were conducted virtually using Zoom [ 52 ]. The key informant interviews and focus groups were audio-recorded and transcribed. All data were aggregated, anonymized, and securely stored in a cloud server. Data are owned by the community. Both the Council and the research team have equal access to the data.

Data analysis

All documents identified through the environmental scan were reviewed for key themes. A list of existing school and community programs was compiled and organized by theme (i.e., education-focused, nutrition-focused, health-focused, etc.). Follow-up conversations with key informants verified the continued planning and provision of these programs.

Following the 6-step method by Braun and Clarke (2006), a thematic analysis was conducted to systematically identify key topic areas and patterns across discussions [ 53 ]. A shortlist of themes was created for the key informant interviews and focus groups, respectively. A manual open coding process was conducted by two reviewers who reached consensus on the final coding manual and themes. Separate analyses were conducted for key informant interviews and focus group discussions; however, findings were synthesized to identify key themes and sub-themes in key priorities for the community, community supports and barriers, as well as digital connectivity and infrastructure needs.

Needs assessment findings

The needs assessment guiding framework informed specific discussions of key issues in the community of Île-à-la-Crosse. Key informant interviews and focus group discussions commenced by asking about priorities–“what are the key areas of focus for the community?” In all conversations–including a document review of initiatives in Île-à-la-Crosse–health was highlighted as a current priority; hence, questions in the guiding framework were tailored to fit a needs assessment focused on community health. The following five overarching evaluation questions were used to guide the evaluation: i) What are the prominent health issues facing residents of Île-à-la-Crosse?; ii) What supports are currently available to help residents address prominent health issues in the community?; iii) What types of barriers do community members face to accessing services to manage their health?; iv) How is health-related information currently shared in the community?; and v) To what extent are health services and information currently managed digitally/electronically? The evaluation questions were kept broad to capture a range of perspectives. An evaluation matrix linking the proposed evaluation questions to their respective sub-questions, indicators, and data collection tools is outlined in Table 1 .

https://doi.org/10.1371/journal.pone.0279282.t001

Feedback on each needs assessment topic area is summarized in the sections below. Sample quotes supporting each of the key topic areas is provided in Table 2 .

https://doi.org/10.1371/journal.pone.0279282.t002

Key priorities

Four priorities were identified through the focus groups, key informant interviews, and document review ( Fig 2 ). Given the timing of the discussion, the primary issue of concern was the COVID-19 pandemic. Many community members were worried about contracting the virus, and the risk it posed to Elders in the community. Of greater concern, however, was how COVID-19 exacerbated many existing health concerns including diabetes and hypertension in the community. For example, routine procedures were postponed and community members with other health conditions were not receiving routine healthcare during the height of the pandemic. The St. Joseph’s Hospital and Health Centre services Île-à-la-Crosse and bordering communities, hence maintaining capacity for COVID-19 patients was a priority. COVID-19 exposed existing barriers in the healthcare system which are described in greater detail in the barriers to community health section.

https://doi.org/10.1371/journal.pone.0279282.g002

Another priority discussed by many community members was climate change and the environment. Community members noted that changes in wildlife patterns, land use, and early winter ice road thaw were areas of concern, particularly due to the impact these factors have on traditional food acquisition practices (i.e., hunting) and food access. For instance, the geographic location of Île-à-la-Crosse is surrounded by a lake, and the main highway which connects the community to the land has experienced increased flooding in the past few years.

In addition to posing immediate danger to community members, food security and sovereignty are also closely linked to road access. While the community produces some of its own food through the local fishery and greenhouses, Île-à-la-Crosse is still dependent on a food supply from the south (i.e., Saskatoon). During COVID-19, food access was further restricted due to limited transport and delivery of food products, which increased the risk of food insecurity for community members. Food insecurity was believed to be of bigger concern for Elders in the community compared to younger members. Younger community members expressed having the ability to source their own food in a variety of capacities (e.g., fishing in the lake), whereas Elders rely more heavily on community resources and support (e.g., grocery stores, friends, and family).

Community members also discussed issues surrounding mental health and wellbeing. This topic was of particular concern for youth and Elders in the community. Community members discussed the importance of identifying covert racism (vs. discrimination) that exists within health services that exacerbated mental health issues and care, as well as developing coping strategies, resilience, and supports to prevent mental health crises. Key informants emphasized the need to minimize the stigma around mental health and focus on holistic wellbeing as they work to develop strategies to improve community wellness.

Community health supports

Île-à-la-Crosse has been working on developing supports to improve community health through various initiatives. A document review identified a community-specific wellness model which has informed program development and planning over the past few years. The key components of the Île-à-la-Crosse wellness model are: i) healthy parenting; ii) healthy youth; iii) healthy communities; iv) Elders; v) healing towards wellness; and vi) food sovereignty. The Elders Lodge in the community provides support for holistic wellbeing by promoting intergenerational knowledge transmission, guidance to youth and community members, as well as land-based activities which improve bonding, cultural awareness, and mental and spiritual well-being among community members. The Elders Lodge hosts both drop-in and organized events.

Several initiatives have been developed to support food sovereignty in the community, including a greenhouse program where fruits and vegetables are grown and shared locally. This program is run in partnership with the school to increase food knowledge and skills among youth. In addition, after-school programs including traditional food education (i.e., cooking classes) and land-based activities (i.e., berry picking) led by Elders support the goals of the wellness model. The community is currently working on developing additional programs dedicated to improving mental wellness among adults, youth, and Elders.

Barriers to community health

When key informants were asked to identify barriers to community health, they described delays in access to timely health information. For example, daily COVID-19 tests conducted at the regional health centre in Île-à-la-Crosse were relayed to the provincial health authority; however, information about the total number of COVID-19 cases could take up to one week to be sent back to the community. This time lag restricted community decision-makers’ ability to enact timely policy (i.e., contact tracing) and rapidly respond to managing cases.

A second barrier that was raised by community members was a delay in access to timely healthcare. The Île-à-la-Crosse hospital is a regional health service centre serving the community as well as surrounding areas. Community members noted that the load often exceeded the capacity of the single hospital, and some patients and procedures were relocated to hospitals and clinics in the larger city of Saskatoon, Saskatchewan. This was reported to be challenging for many community members as it was associated with longer wait times, long commutes, and sometimes required time off work. Many of these challenges were exacerbated during the COVID-19 pandemic. As a result of the pandemic, many medical centres and hospitals postponed routine and elective medical procedures in an attempt to accommodate the overwhelming influx of patients who contracted COVID-19. In addition, community members were advised to avoid spending time in health centres to limit risk of exposure to the virus. These COVID-related changes further delayed access to timely healthcare for many community members of Île-à-la-Crosse.

Several community members reported experiencing institutional racism in healthcare and social service settings outside of Île-à-la-Crosse. This was particularly exacerbated during the COVID-19 movement restrictions, where community members faced significant difficulties in accessing services and care in larger urban centres, and experienced further discrimination due to the stigma of COVID-19-related rumours about communities in the north.

Lastly, community members discussed a lack of awareness about some health topics, including where and how to access reliable health information. Some community members attributed this lack of awareness to a general distrust in government health information due to a history of colonialism and exploitation in Canada, which likely contributed to increased misinformation about COVID-19 risk and spread.

Health communication

The primary modes of communication within Île-à-la-Crosse are radio and social media. These platforms were used throughout the pandemic to communicate health information about COVID-19 case counts and trends. Community members also reported obtaining health information from healthcare practitioners (i.e., for those already visiting a healthcare provider), Elders, and the internet. Key informants indicated an interest in improving digital infrastructure to enable sharing of timely and accurate health information with community members and minimize misinformation. Key informants also reported room for improvement in the community’s digital health infrastructure, particularly in improving timely communication with community members, and to inform decision-making in crisis situations.

Digital infrastructure and connectivity

Île-à-la-Crosse has its own cell tower which offers reliable access to cellular data. The community also has access to internet via the provincial internet provider–SaskTel, as well as a local internet provider—Île-à-la-Crosse Communications Society Inc. Key informants and community members confirmed that most individuals above 13 years of age have access to smartphones, and that these mobile devices are the primary mode of internet access. However, it was unclear whether everyone who owns smartphones also has consistent data plans or home internet connections. Key informants described the great potential of digital devices like smartphones to increase the speed and accuracy of information sharing. Discussions with both key informants and community members suggested the need for a community-specific app or platform which could provide timely health information that was tailored to the community’s needs.

Community members noted that expanding digital infrastructure had to be paired with efforts to improve digital literacy–particularly as it relates to data security, privacy, and online misinformation. A separate initiative was discussed which could work to improve digital literacy among youth and Elders, as this would improve both the uptake of digital health platforms, as well as their usefulness and application. Key informants discussed the importance of building digital infrastructure that would enable data sovereignty, self-governance, and determination. The key informants, who are also primary decision-makers in the community, described opportunities for ethical development of digital platforms that would ensure that data is owned by the community.

Needs assessments are commonly the first step in understanding specific community needs, [ 27 , 28 ]; however, few evaluation frameworks provide practical guidance on how to engage communities in needs assessments [ 41 ]. This paper provides a step-by-step guide for conducting needs assessments in collaboration with communities in the digital age. Using the series of questions outlined in the Guiding Framework, researchers and evaluators can gain an in-depth understanding of a community’s priorities, needs, existing capacity, and relevant solutions.

The Guiding Framework was critical to establishing a partnership with the community of Île-à-la-Crosse, as it enabled the research team to obtain detailed insight into their priorities–in this case, community health–as well as community capacity. Taking a Two-Eyed Seeing approach [ 15 ], conversations with the community highlighted strengths of Western digital technology and the diversity of Indigenous Knowledges for addressing priorities [ 13 ]. This approach was also important to establishing trust and respect for the variety of perspectives that could be used to address community priorities. The resulting partnership also enabled the conceptualization of tangible action items that were aligned with current and future priorities–a key factor in the sustainability and feasibility of community-based initiatives [ 4 – 8 , 54 ].

Challenges and opportunities for using digital platforms for priorities identified by needs assessment

Many rural and remote communities face similar challenges and share common priorities with Île-à-la-Crosse. For example, resource and service access, including food and other essential supplies, healthcare, and internet connectivity are issues faced by many rural and remote communities across Canada [ 55 – 60 ]. Key informants and community members from our partner community corroborated these access issues, particularly in relation to public health. Given the potential for digital technology to bridge access gaps, it has become pertinent to invest in digital infrastructure and platform development.

Research has shown that in many rural and remote communities, smartphone ownership is not the limiting factor–it is internet inequity, which is defined as differential internet access based on wealth, location (urban, rural, or remote), gender, age, or ethnicity [ 61 ]. The United Nations has declared internet access a human right [ 10 ], which makes it imperative to develop digital infrastructure such as internet connectivity to improve digital accessibility. Île-à-la-Crosse has its own cell tower which offers reliable access to cellular data. The community of Île-à-la-Crosse also has access to consistent and dedicated internet service through a provincial internet provider and local internet provider. The needs assessment showed that the universality of smartphone ownership combined with good internet connectivity lays the foundation for the development of tailored, culturally appropriate digital health platforms in communities like Île-à-la-Crosse.

In particular, the needs assessment revealed that smartphone apps, which most citizens are well-versed with, can be used to provide local services and access to resources. For example, a locally developed app can connect the Mayor’s office with community members in real-time to provide updates on COVID-19 outbreaks. Apps also have the potential to connect communities to resources within and outside of the community [ 35 , 57 ]. For example, advanced artificial intelligence algorithms can be used to anticipate community needs prior to urgent crises like COVID-19, environmental disasters, or food crises [ 35 , 62 – 65 ]. To date, the issue has not been the lack of technology or ability to bridge this gap for rural and remote communities. Instead, larger systemic inequities have limited our ability to co-create local solutions for global problems by decentralizing technology that is widely available [ 35 , 66 ], which highlights upstream inequities in developing digital platforms.

Recommendations for inclusive digital needs assessments

Given the widespread adoption of digital technology, digital platforms can provide rich data to identify and address community crises [ 2 , 3 , 35 ]. Importantly, co-created digital platforms can be used to share knowledge in real-time with community members and other stakeholders to enable remote engagement, which is especially important during crisis situations such as a pandemic [ 2 , 3 , 35 ]. As we implement creative digital platforms in varied programs or research projects, we must also integrate this digital perspective into the evaluation process. Research and evaluation literature has well established approaches to needs assessment evaluations [ 29 , 42 , 67 ]; however, in the 21st century, we need to account for the use and application of digital platforms in community-focused initiatives. To identify how and where digital platforms can play a role in addressing community priorities, we propose several recommendations for inclusive community-based needs assessments.

First, at the crux of all community-based needs assessments is relationships. A relationship built on respect, reciprocity, mutual understanding, and prioritizing the needs and vision of communities is essential for sustainable impact. The First Nations OCAP® principles [ 68 ] informed conversations between the research team and community about data ownership and control. These principles include ownership of knowledge and data, control over all aspects of research, access to information about one’s own community, and possession or control of data [ 68 ]. The OCAP® principles ensure First Nations and other Indigenous Peoples the right to their own information, and also reflect commitments to use and share information in a way that maximizes the benefit to a community, while minimizing harm. Some communities may choose to lead a project, or work closely in collaboration with experts for specific projects. Irrespective of the project dynamics, needs assessments rely on detailed information and context about a community for a project to succeed.

Second, it is important for researchers and evaluators to gain an understanding of the current digital infrastructure and connectivity in the community. The needs assessment framework ( Fig 1 ) includes relevant questions for identifying data and WIFI access in a community, penetration of digital devices, and existing digital infrastructure. Even for community-based initiatives that are not focused on a digital platforms, digital technologies will inevitably be a part of the solution, a barrier, or both. Hence the digital landscape has become part of the context that we must capture and understand in a needs assessment to better design and develop programming, policies, and other initiatives.

Third, it is important to ask the question of where and how a digital tool or platform could help. Are there gaps that digital platforms can help address or fill? In rural and remote communities, in particular, digital platforms can provide access to real-time information and services not otherwise available. For example, Telehealth [ 69 , 70 ] in the Canadian north offers citizens access to essential healthcare services, including video appointments with medical specialists. Prior to Telehealth, many residents would need to fly into bigger cities in the nearest province to access health care [ 55 ].

Lastly, an understanding of the broader context which affects a community’s ability to adopt digital platforms is critical to the success of digital initiatives. This includes, but is not limited to, capturing data on socioeconomic status and the accessibility of internet-connected digital devices. Digital platforms should help to bridge the divide in resource, service, and information access–not widen the gap. For some communities, this may require working on building digital infrastructure and obtaining dedicated funds to expand access prior to implementing digital initiatives. In addition, digital literacy cannot be taken for granted. Digital literacy refers to individuals’ ability to not only use digital devices, but according to Eshet-Alkalai [ 71 ], “includes a large variety of complex cognitive, motor, sociological, and emotional skills, which users need in order to function effectively in digital environments.” In its simplest form, digital literacy may include the ability to navigate digital platforms, download apps, and communicate electronically. Other more specific skills include ability to read and understand instructions, terms and services, as well as data privacy and security statements [ 72 – 74 ] As part of a needs assessment, identifying digital literacy within a community is an important step to safe, ethical, and relevant digital tool development.

Considering the challenges, immense potential, and learnings from applying the Guiding Framework, a tailored digital platform was conceptualized called Sakitawak Health.

Development of Sakitawak Health

Sakitawak Health is a culturally-responsive digital epidemiological platform to monitor, mitigate, and manage COVID-19 outbreaks. The needs assessment concluded that digital platforms can be used for emerging or other existing population health crises within Île-à-la-Crosse and potentially other Indigenous communities. Moreover, to co-create digital platforms, the Île-à-la-Crosse Citizen Scientist Advisory Council identified key features to embed in CO-Away, including free virtual care for citizens via a smartphone app at the frontend, and access to anonymized community data on the backend for decision-makers.

The app will provide three key precision medicine services that are specific to each citizen: 1) continuous risk assessment of COVID-19 infection; 2) evidence-based public health communication; and 3) citizen reporting of food availability, access to public services, and COVID-19 symptoms and test results. These culturally-responsive features have been co-created with Métis decision-makers in Île-à-la-Crosse based on imminent community needs and preferences. CO-Away will enable real-time data collection through continuous citizen engagement to inform municipal jurisdictional policies.

There are three guiding principles for developing Sakitawak Health: I) Citizen empowerment and data ownership: Active engagement is enabled through app features such as visualizing community risk. More importantly, the community owns the data to ensure data sovereignty; II) Privacy: Utilizing a cutting-edge methodology called federated machine learning, we will develop artificial intelligence algorithms that stores sensitive data such as participant location on mobile devices itself (i.e., sensitive data are not stored in external servers); III) Security and scalability: The backend server will be located in Cloud in Canada, which allows for horizontal and vertical scalability (i.e., the potential for developing multiple frontend apps and decision-making dashboards).

Recognizing the importance of data sovereignty and Indigenous self-governance

Data sovereignty and social justice are important aspects of community-based work, particularly for communities that have experienced discrimination or systemic inequities [ 2 , 75 ]. Data sovereignty refers to meaningful control and ownership of one’s data [ 76 ]. For Indigenous communities in Canada, self-determination and self-governance are of paramount importance given the colonial history of oppression, trauma, and disenfranchisement [ 77 ], and data sovereignty and ownership of digital platforms can promote that independence. In conducting digital community-based needs assessments, the application of a Two-Eyed Seeing lens enables us to leverage strengths of both Indigenous and Western Ways of Knowing to help focus on key priorities and develop solutions.

The engagement and overall approach to working with the community of Île-à-la-Crosse applied a Two-Eyed Seeing lens. In the needs assessment with Île-à-la-Crosse, Two-Eyed Seeing involved incorporation of Métis Knowledge during team engagements, which ensured that any digital platforms developed would incorporate Indigenous Knowledge to promote data sovereignty. All priorities identified within this manuscript are a result of direct Indigenous Knowledge that was provided by the Council. Indigenous Knowledge is not limited to the knowledge of Elders and Traditional Knowledge Keepers; however, they play a critical role in guiding that knowledge through by providing historical, geographic, and cultural context. Discussions with Île-à-la-Crosse about data sovereignty centered around citizen ownership of data, community access, and ensuring data privacy and security. The ultimate goal of this approach to data sovereignty is to facilitate decreased dependence on external systems and use digital solutions for Indigenous self-determination and self-governance.

The needs assessment represents the first phase of a larger evaluation strategy to develop and implement culturally appropriate digital platforms for community health. Phase 1 involved identifying core health priorities and desired supports in the community of Île-à-la-Crosse. Based on the needs assessment findings, Phase 2 of this project will involve the development of tailored digital health platforms and programming to support digital literacy. As part of Phase 2, digital literacy programs and tailored digital health platforms will be pilot tested and adapted prior to their implementation. In Phase 3, a process evaluation will be conducted to assess the reach, uptake, and use of digital health platforms and digital literacy programming. Integrated knowledge translation will be conducted during all phases to ensure continuous feedback, communication, and knowledge sharing with all relevant stakeholder groups.

Conclusions

Needs assessments can facilitate important conversations in community-based research and evaluation to learn about key priorities, challenges, and opportunities for growth. The Guiding Framework for Community-Based Needs Assessments to Embed Digital Platforms details a step-by-step approach to begin a conversation with communities to better understand their needs, and to tailor research and evaluation projects focused on embedding digital platforms. In Île-à-la-Crosse, the needs assessment framework has propelled the launch of a timely, community-engaged digital initiative to address key priorities, starting with COVID-19. Overall, tailored platforms can help bridge existing gaps in resource, program, and service access in Indigenous communities, irrespective of their location across the world.

Supporting information

https://doi.org/10.1371/journal.pone.0279282.s001

Acknowledgments

The authors would like to acknowledge the contributions of community members of Île-à-la-Crosse. The Elders, youth, and key decision-makers who are part of the Île-à-la-Crosse Citizen Scientist Advisory Council have been invaluable in providing support, guidance, and cultural training to the research team. The authors also acknowledge the support of the Canadian Internet Registration Authority in advancing the uptake of digital health applications.

• View Article
• Google Scholar
• 2. Smith LT. Decolonizing Methodologies: Research and Indigenous Peoples. London; New York; Dunedin: Zed Books Ltd; University of Otago Press; Distributed in the USA exclusively by St Martin’s Press; 2021. https://doi.org/10.4324/9781315597843-10 .
• 7. Narayan D. Empowerment and Poverty Reduction: A Sourcebook. Washington, D.C.; The World Bank; 2002.
• 8. Cairncross L, Morrell C, Drake J, Brownhill S. Tenants Managing: An Evaluation of Tenant Management Organisations in England. London: Office of the Deputy Prime Minister; 2002.
• 9. Pratchett L, Durose C, Lowndes V, Smith G, Stoker G, Wales C. Empowering communities to influence local decision making: systematic review of the evidence. London: 2009.
• 10. La Rue F. Report of the Special Rapporteur on the Promotion and Protection of the Right to Freedom of Opinion and Expression, Frank La Rue: addendum. 2011.
• PubMed/NCBI
• 19. Statistics Canada. Île-à-la-Crosse, NV [Census subdivision], Saskatchewan and Saskatchewan [Province] (table). Ottawa: 2017. https://doi.org/Statistics Canada Catalogue no. 98-316-X2016001.
• 20. Waldram JB, Herring A, Young TK. Aboriginal Health in Canada: Historical, Cultural, and Epidemiological Perspectives. 2nd editio. Toronto: University of Toronto Press; n.d.
• 22. Central Urban Métis Federation Inc., Kinistin Saulteaux Nation, Saskatoon Health Region. Strengthening the Circle: Partnering for Improved Health for Aboriginal People. Saskatoon, SK: 2010.
• 28. Pavlish CP, Pharris MD. Community-based collaborative action research : a nursing approach. Sudbury, MA: Jones & Bartlett Learning; 2012.
• 42. Stufflebeam DL, Shinkfield AJ. Evaluation theory, models, and applications. 1st ed. San Francisco, CA: Jossey-Bass; 2007.
• 43. McLennan D. Sakitawak Development’s Facebook Page n.d.
• 44. Cameco Corp. Patuanak—English River First Nation—Community Profiles—Community. Cameco North Saskatchewan 2020.
• 45. McLennan D. The Encyclopedia of Saskatchewan: Île-à-la-Crosse n.d. https://esask.uregina.ca/entry/ile-a-la-crosse.jsp .
• 52. Zoom Video Communications Inc. Security guide 2016.
• 68. The First Nations Information Governance Centre. The First Nations Principles of OCAP TM Training Course 2021:1–6. https://fnigc.ca/ocap-training .
• 69. Telehealth 2021. North Heal 2021. https://www.northernhealth.ca/services/digital-health/telehealth .
• 75. Kukutai T, Taylor J, editors. Indigenous Data Sovereignty: Toward an Agenda. The Australian National University Press; 2016. https://doi.org/10.1109/IPDPSW.2014.214 .

Needs Assessments Processes, Methods and Examples: Annotated Bibliography

Western region evaluation network (wren).

• University of Alaska, Fairbanks; Alda Norris
• University of Arizona; Michele Walsh
• University of California, Davis; Vikram Koundinya
• Colorado State University, Cary Weiner
• University of Idaho; Nav Ghimire
• Montana State University; Steven Siegelin
• University of Nevada, Reno; William Evans
• New Mexico State University; Lajoy Spears
• Utah State University; Lendel Narine
• Washington State University; Rebecca Sero
• University of Wyoming; Kim Reaman

Introduction

One of the first priorities for a new Extension educator or agent is to conduct a needs assessment to inform their programming (Caravella, 2006). Needs assessment involves ascertaining the current circumstances and understanding what is desired in the future, and comparing the two (Altschuld & Watkins, 2014).  Altschuld and Watkins (2014) further state that needs assessment also includes judgments related to prioritizing the identified needs to guide programming and planning decisions. Angima, Etuk and King (2014) state that a solid needs assessment is the foundation of a successful Extension program. Needs assessment is an essential step in the program planning, development and evaluation cycle (Etling & Thomas, 1995). Needs assessment also has been identified as one of the key educational competencies for Extension educators (Ghimire, 2010; Koundinya, 2010). The value of needs assessment for Extension programming relies on its ability to succesfully identify stakeholder needs, and be empowered with the necessary information to design programs, products and services to meet those needs (Garst & McCawley, 2015).

Many state Extension systems require Extension faculty to conduct a systematic and thorough needs assessment, and document the results in their reviews. In this context, it is important to have an annotated bibliography focused on needs assessment processes and methods with relevant examples for Extension professionals. This annotated bibliography is Extension-centric with examples from other state Extension systems and community-based needs assessments related to Extension programming.The first section has studies related to processes, methods and types of needs assessments. The second section contains examples of needs assessments of various types, scopes and focuses. This bibliography is part of a larger set of educational and training resources on needs assessment, program development, program evaluation, and impact writing and storytelling developed by the Western Regional Evaluation Network (WREN).

Needs Assessment Processes and Methods

Batsche, C., Hernandez, M., & Montenegro, M. C. (1999). Community needs assessment with Hispanic, Spanish-monolingual residents . Evaluation and Program Planning, 22(1), 13-20 .

The authors argue for the importance of needs assessment to ensure representation of the diverse population that live in the community. The authors describe methods to increase participation of Hispanic residents in community needs assessment activities, based on their experience with needs assessment activites in Tampa, Florida. These suggestions are in regards to: 1. Defining the population, 2. Estimating the population size, 3. Selecting the sample, 4. Translating the survey, 5. Gaining access to respondents, 6. Designing and administering the survey, 7. Timing of interviews, 8. Location of interviews, 9. Interpreting the results, 10. Summary of findings, 11. Disseminating the findings, and 12. Strength-based assessment.

Bridges, C. (2008). Identifying agriculture and forestry educational needs using spatial analysis techniques. Journal of Extension, 46(3), 3TOT6 .

The author argues for the use of spatial analysis techniques to identify agriculture and forestry educational needs. The author argues that needs assessments are important to prioritize educational programs toward producers, and that given the need for onsite instruction in agricultural education, understanding spatial variation in agriculture is key. This article presents a simple method to combine needs assessments with spatial analysis techniques such as Geographic Information Systems. The author argues that this method has potential to identify changes in agricultural practices and to prioritize where Extension efforts should be targeted.  

Caravella, J. (2006). A needs assessment method for Extension educators. Journal of Extension, 44(1), 1TOT2.  

This article presents various needs assessment methods used by the family living Extension agent in rural Wisconsin. Needs assessments or situational analysis are presented as the first step in an ongoing plan to evaluate the local environment in which Extension functions to identify priority program focuses. The article describes the methods used in this needs assessment, which include: census data, existing local needs surveys and conducting interviews with key informants (n= 25). The key informants included people from county government departments, law enforcement, the court system, not-for-profit agencies and religious organizations.

Donaldson. J., & Franck, K. (2016). Needs assessment guidebook for Extension professionals. The University of Tennessee Extension Publications. PB 1839 . 

This publication is a guidebook on how to organize needs assessments. It provides details of the three phases of conducting an effective needs assessment: exploration, assessment, and utilization. Different methods for each phase are provided, such as document reviews, individual and group methods (such as key informants, personal interviews, questions for interview, or committees, focus groups, surveys, open listing sessions, brainstorming, concept mapping, among others), interpretation of results and communicating the results of the needs assessment. Each section contains useful tools to design, conduct and interpret effective needs assessments.

Ekins, J. (2018). Extension involvement in collaborative groups: An alternative for gathering stakeholder input. Journal of Extension, 56(2). 2IAW5 .

The author proposed an alternative for community educational needs assessment to focus group research, which consists of participant observation research with collaborative stakeholder groups. The author compares focus group research with collaborative stakeholder group observation in different dimensions, provides different examples of published work with this method, and concludes that the method provides a robust assessment, and in addition, offers great networking opportunities and collaborative sources of information.

Ennis, G., & West, D. (2010). Exploring the potential of social network analysis in asset-based community development practice and research. Australian Social Work, 63:4, 404-417 . 

This article makes the case for the incorporation of key concepts from social network theory on asset-based community development practice. The authors propose that the focus should be on community “assets” instead of “needs,” and that social network analysis is a methodology for understanding the efficacy of asset-based community development (ABCD) projects. The authors also argue that currently, although the asset-based community development model is widely used, it is criticized for its lack of an evidence base, theoretical depth and consideration of macro-level causes of disempowerment. Social network concepts have been rarely used in a comprehensive manner in asset-based community development practice or research. This paper proposes to address these challenges through: 1. Exploring the strengths and limitations of ABCD, 2. Broadly overviewing social network theory analysis, and 3. Considering its potential to be integrated into ABCD practice and research. The paper concludes by arguing for the potential of social network analysis as a framework to study ABCD interventions and to understand how ABCD interventions impact the different elements of a community.

Folinsbee, S., & Jurmo, P. (1994). Collaborative needs assessment: A handbook for workplace development planners. ABC Canada . 

This handbook outlines a four-phase collaborative approach to assessing a work organization’s education needs and specifying a range of activities to meet the identified needs. Such phases are: 1. laying the groundwork for a workplace needs assessment (get all interested groups committed, organize a planning committee, plan and hold initial committee meetings, and design the workplace needs assessment); 2. Carrying out the needs assessment (prepare to gather information, collect information in personal interviews and focus groups, gather information from other sources, organize the information, and analyze selected workplace documents); 3. Interpreting and reporting (interpret the information, write the final report, and report the findings); and 4. Deciding what happens next (develop and get commitment for an action plan and evaluate the workplace needs assessment).

Garst, B., & McCawley, P. (2015). Solving problems, ensuring relevance, and facilitating change: The evolution of needs assessment within Cooperative Extension. Journal of Human Sciences and Extension, 3(2), 26-47 . 

The authors trace the history and evolution of needs assessments within Cooperative Extension. They first define the goal of needs assessments as twofold: 1. to understand stakeholders’ problems,  and/or concerns, and 2. to understand how to respond with program, products and services. The authors also present some reasons for conducting needs assessments (adapted from Etling & Maloney, 1995), which include: program planning, principle of democracy, motivation, accountability, support, anticipation of conflicts, needs change and complex society.

The authors also describe the historic evolution of needs assessments, where they highlight how needs assessments have been used over the decades in Extension. Before the 1960s, needs assessments were conducted to identify and prioritize programs for clientele. Local groups helped plan how to meet objectives rather than identifying priorities. Between the 60s and 70s, Extension services adopted more sophisticated protocols to evaluate if programs were meeting the needs of beneficiaries. During the 80s, there was a significant growth in the use of needs assessment practices, but given budget cuts, more cost-effective methods emerged. During the 90s, a more integrated approach was adopted for needs assessments, incorporating analysis of secondary data and conversations with key informants. Kaufman’s Organizational Elements Model (OEM)  was solidified, and asset mapping was introduced in this decade, which helped with capacity assessment. Finally, during the 2000s to the present, new technologies were introduced for needs assessments, such as new information technology platforms, Geographic Information Systems (GIS), real-time data collection, visual display and data storage in an interactive setting. Participatory research approaches to needs assessments were disseminated. Photovoice techniques also emerged.

Guion, L. (2010). A 10-step process for environmental scanning. Journal of Extension, 48(4), v48-4iw2 . 

This article describes a 10-step process for conducting environmental scanning in North Carolina Cooperative Extension (NCCE). The author defines environmental scanning as a study of emerging forces within an organization (as cited by Boone, 1992; Boone, Safrit, & Jones, 2002), and highlights it as an opportunity to explore diverse sources of information and viewpoints to accomplish Extension goals and find educational solutions. The 10 steps are: 1. The county team conduct situational analysis using secondary data, 2. List issues that are important, based on secondary data, 3. Conduct situational analysis using primary data from major stakeholder groups, 4. Map county to obtain primary data from a cross section of the population, 5. Collect primary data in each of the mapped areas, 6. List issues that consistently surfaced as important in Steps 2 and 3, 7. Conduct external asset assessments, 8. Prioritize issues, 9. Examine the complexity and interdisciplinary nature of priority issues, and 10. Each county enter priority issues, internal and external assets, and the integrated programming strategies to address the issues into the web-based NCCE county priority issues database. The article describes each of these 10 steps and provides a practical guide for conducting environmental guides at the county level.

Havercamp, M., Christiansen, E., & Mitchell, D. (2003). Assessing Extension internal organizational needs through an action research and learning process. Journal of Extension, 41(5), 5FEA2.  

This article describes how a participatory action research and learning process were used for a needs assessment of an Extension organization. The article describes the methods used, which included focus groups, questionnaires and employee feedback sessions. The goal was to identify the internal needs of University of Nevada Cooperative Extension (UNCE) over a five-year period for strategic planning. The authors argue that the results of this study have been used by UNCE, and recommend the use of participatory action research and learning process for conducting internal organizational needs assessments. The authors highlight the importance of engaging different personnel (Extension leaders, faculty and staff) when implementing the assessment.

Hillier, A. (2007). Why social work needs mapping. Journal of Science Work Education, 43(2), 205-222.  

The author presents several reasons why social work would benefit from the use of GIS. Among such reasons, the author argues that GIS is critical to identify community needs and assets. Mapping allows understanding of how the environment impacts individuals and provides evidence of disparity. The author provides examples of social work using GIS that has shown high levels of spatial inequality among recipients who tended to live in areas with poor service, providing evidence of a spatial mismatch. The author argues that mapping allows information about individuals and households to be integrated with information about their communities, so that funders, service providers and researchers can understand individuals in the context of their communities. The author makes the case that documenting needs is not enough, but that needs have to be spatially located. For the author, the integration of GIS in social work will improve delivery of social services and empower communities and traditionally disenfranchised groups.

Kaplan, M., Shih-Tsen, L., & Radhakrishna, R. (2003) Intergenerational programming in Extension: Needs assessment as planning tool. Journal of Extension, 41(4), 4FEA5.  

The authors present the findings of their needs assessment study conducted to plan and develop a statewide intergenerational program. The authors conducted a mail survey, directed to Extension educators in Pennsylvania. Respondents noted their preferences on program content and delivery format. The results were used to make decisions about curricular directions and program delivery strategies. These results and their incorporation in planning strategies showcase the value that needs assessments provide for setting program priorities and the importance of assessing the perceptions of staff delivering programs.  

Kenneth, L. C. (1993). Needs assessment for group work with people of color. Social Work with Groups, 15:2-3, 53-66 . 

The author claims that cultural sensitivity in needs assessment is necessary for effective group work practice with ethnically diverse populations. The author introduces a conceptual framework to assess the nature of psychological needs borne out of cultural differences. This method uses “sociocultural dissonance” as an orienting concept, which is based on three major categories of psychological needs: life-change events, role-status change, and social-structural adjustment. The author discusses how each of these dimensions is manifested at the individual, intragroup, intergroup and systems level. Finally, the paper considers implications for group practice.

Kerka, S. (2003). Community asset mapping. Educational Resources Information Center (ERIC). Trends and Issues Alert, 47 . 

The author describes asset mapping as a method for documenting the tangible and intangible resources of a community, where assets may be persons, physical structures, natural resources, institutions, businesses or informal organizations (as cited by Berkowitz & Wadud, 2003). The asset-mapping approach draws on appreciative inquiry, recognition of social capital, participatory approaches to development, collaborative economic development models and efforts to strengthen civil society (as cited by Mathie & Cunningham, 2002). In this note, the author lists and describes resources that can help community educators use asset-mapping approaches for program planning.

Kramer, S., Amos, T., Lazarus, S., & Seedat, M. (2012). The philosophical assumptions, utility and challenges of asset mapping approaches to community engagement. Journal of Psychology in Africa, 22:4, 537-544.  

This article presents a literature review and discussion of different asset-based mapping approaches, such as: Asset-Based Community Development (ABCD); Participatory Inquiry into Religious Health Assets, Networks and Agency (PIRHANA); Community Health Assets Mapping for Partnerships (CHAMP); the Sustainable Livelihoods Approach (SLA); Planning for Real®; and approaches using Geographic Information Systems (GIS). The review describes and examines these approaches to show ways in which asset-focused approaches as interventions could be used to promote community development through engagement. The authors conclude that these methods of asset-based mapping are valuable because of their key capacities for partnership building, consensus creation, and community agency and control. The authors suggest that when applied, these methods should ensure context-specificity, facilitate co-learning, encourage resource sharing and promote community empowerment.

Lightfoot, E., Simmelink, J., & Lum, T. (2014). Asset mapping as a research tool for community-based participatory research in social work. National Association of Social Workers.  

This research note describes the asset-mapping approach as a community-based participatory research (CBPR) method for social work. The article describes its origins, provides an overview of asset mapping and presents it as a research technique and as a method of asset mapping in CBPR and in social work research. The authors conclude that asset mapping is well suited for social work practice and as a research method for use in social work CBPR research partnerships that explore social issues. It can help tease out the strengths of communities, fits well with social work’s strengths-based approach to conducting research in low-resource communities, and is a fairly straightforward approach. In addition, the evidence base for Asset-based Community Development (ABCD) projects is robust.

Lien, A., Ruyle, G., & López-Hoffman L. (2018). Q methodology: A method for understanding complex viewpoints in communities served by Extension. Journal of Extension, 56(2), V56-2iw4 .

This article introduces Q methodology by explaining what it is and how it works, and providing an example of its use. Q methodology is an activity to understand individual viewpoints, and sort ideas. The method uses a card-sorting exercise for researchers to analyze individuals’ subjective viewpoints on different issues and groupings of different viewpoints within a community. The difference between Q methodology and focus groups, surveys or interviews, the authors claim, is that Q methodology focuses on the participant, not the participant’s answers. It provides an understanding of how respondents think about the questions themselves, rather than how respondents answer the questions about different aspects of an issue. The article explains the methodology and provides a didactive example of how the method works, as well as recommendations for further readings. This method provides an in-depth understanding of an issue and of the context that informs the subject’s thinking around the issue. It is of use for Extension, as it could help for conflict mediation or resource management.

North Carolina Cooperative Extension. 2017-2018 Needs assessment community statewide summary. NC State Extension.  

North Carolina Cooperative Extension provides several resources for county needs assessments, including: how to summarize county data, statewide report examples and methods for data collection, such as: secondary data, different instructions to conduct interviews, county commission surveys, stakeholder focus groups, employee surveys and citizen surveys. In the examples they provide they use multi-methods approaches to collect county data and stakeholder inputs. They also provide on their reports detailed examples for the data analysis and how to visualize most important community issues for relevant questions.

Phibbs, E., Relf, D., Hunnings, J. (2005). Implementing a needs assessment for long-term strategic planning in 4-H horticulture programming. Journal of Extension, 43(4), 4RIB7 .

This article describes a needs assessment implemented for 4-H horticulture programming to be used for strategic planning by Virginia Cooperative Extension (VCE). The needs assessment focused on the satisfaction of clientele with VCE 4-H horticulture publications, programs and available resources. The authors interviewed 4-H agents, agriculture and natural resource agents, Master Gardener coordinators, and 4-H camp directors. The results highlight areas for potential improvement, related to improved communication and resource sharing. The authors argue for the use of needs assessment as an example of strategic planning tool that might be useful for Extension programs.

Singletary, L., & Powell, P. (2003). Conducting a formal needs assessment: A five-step survey approach. University of Nevada Reno . 

This document presents a five-step approach to conducting a formal needs assessment through mail surveys. The five steps are: 1. Develop a broad set of questions and determine protocol, 2. Complete training (IRB) and seek certification to conduct research, 3. Seek IRB approval to conduct survey, 4. Implement needs assessment, record responses and analyze the data (Excel and SPSS and two software programs recommended for the analysis), and 5. Publish an Extension fact sheet, special publication or bulletin to share the results of your needs assessment.

Smith, C., & Freeman, R. (2002). Using continuous system level assessment to build school capacity. American Journal of Evaluation, 23(3), 307-319.  

In this paper, the authors introduce a conceptual framework for internal assessment called Continuous System Level Assessment (CSLA). This method has three phases: needs assessment and problem identification, designing interventions and building staff capacity, and implementing and evaluating interventions. The method is a holistic approach that allows schools to evaluate all programs implemented, and fosters local program and evaluation expertise. The authors argue that CSLA can contribute to more sophisticated professional development strategies for schools.

Wang, C., & Burris, M. A. (1997). Photovoice: concept, methodology, and use for participatory needs assessment. Health Education and Behavior, 24(3), 369-387.

The authors discuss the uses and implications of photovoice methods for needs assessments, based on their experience in Yunnan. First, photovoice techniques fuel critical consciousness and collective action. Second, photovoice provides a community-based diagnostic tool to strengthen inadequate theories in which programs are based. Third, they provide powerful means to advocate for increased funding and guide the distribution of money. Fourth, photovoice may enable grassroots constituents to participate in the policy-making process.

Wille, C., Garcia, Z., & Garcia-Pabón, J. (2019). Collaborating across state lines to leverage cultural competency expertise. Journal of Extension, 57(3), 3TOT6.  

The authors identified the need for a statewide Latino cultural competency training for Utah State University (USU) Extension personnel. This article explains a collaboration between USU Extension and Washington State University (WSU) Extension on adapting and customizing a WSU needs assessment tool for USU Extension faculty. The authors found that collaborating with professionals across the state produced many benefits, such as: streamlined resources and use of already developed material, capitalization of peer experience, customization of other state’s tools and resources, and mutual beneficial collaborations. They also found that administrative collaboration is key to successful cross-state collaboration work.

Worker, S., Schmitt-McQuitty, L., Ambrose, A., Brian, K., Schoenfelder, E., & Smith, M. (2017). Multiple methods needs assessment of California 4-H science education programming. Journal of Extension, 55(2), 2RIB4 . 

This article describes a needs assessment carried out to evaluate 4-H programs on program development and design, professional development, curricula, evaluation, partnerships, and fund development. The results identified areas for growth and needs for improved communication and resource sharing. Multiple qualitative data sources were used (an online open-ended survey, focus group interviews, participatory working groups, data analysis and data integration). The needs assessment revealed opportunities for more intentional and systematic 4-H science programming. It also revealed the need for national and state efforts that provide practical program models and examples of programs targeting scientific literacy, professional development for staff, and consistency in messaging and branding. The authors highlight that using more than one method to analyze and interpret data improved their understanding of the needs and gaps of the 4-H science programming.

Examples of Needs Assessments and Their Use

Duncan, S., & Marotz-Baden, R. (1999). Using focus groups to identify rural participant needs in balancing work and family education. Journal of Extension, 37(1), 1RIB1 . 

The authors illustrate how they used marketing techniques to understand the target audience for program development. Focus groups were carried out with the objective of learning the needs of rural residents in Montana. The authors claim that this study serves as an example of how Extension personnel can gain information about how a program should be produced, priced and promoted, as well as where it should be held to attract the largest number of participants and meet the needs of the target population.

Duttweiler, M. (2008). The value of evaluation in Cooperative Extension. In: Braveman, M. T., Engle, M., Arnold, M. E., & Rannekamp, R. A. (Eds). Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation . 

The author examines how evaluation practices have been of value in Cooperative Extension. The author examines more than 675 evaluations published between 1998 and 2007, through literature reviews, examinations of cases and correspondence with respective authors. The evaluations are characterized by their evaluation purpose and by Jacob’s Levels of Evaluation (Jacobs, 1988). Most evaluations analyzed were related to educational methods improvement, accountability studies or needs assessments. Almost 70% of studies addressed program improvement. The author also finds that most evaluations yielded substantial program modification.

Fuller, J., Bentley, M., & Shotton, D. (2001). Use of community health needs assessment for regional planning in country South Australia. The Australian Journal of Rural Health, 9, 12-17 .

This paper documents how community health needs assessments were used for regional and country health service planning in South Australia between 1995 and 1999. Both local and regional needs assessments were included. Data from needs assessments were translated into health promotion and early intervention program priorities through strategic planning. The authors derive two key lessons of using needs assessments for planning. First, local needs assessments involve local commitment to service change, but are usually slow processes. Second, needs assessments are more likely to be effective if focused rather than broad brushed. 

Hilton, J., Martin, S., & Evans, W. (2007). Meeting the needs of Nevada’s older adults: The role of University of Nevada Cooperative Extension. University of Nevada Cooperative Extension. Educational Bulletin, EB-07-02.  

Nevada’s rapid increase of its senior population is associated with different social processes, such as: increased longevity, higher education of baby boomers who tend to delay retirement, higher standards of living, geographic mobility, greater diversity in family structures, and ethnic composition of the population. A team of University of Nevada Cooperative Extension specialists used several strategies to assess the need for programs for older adults. The team reviewed data and theories, surveyed participants of conferences on aging, assessed services currently provided, and held community forums. These strategies were followed to: 1. Assess the present and future needs of older adults, 2. Investigate the resources available to meet these needs, and 3. Evaluate the gaps between identified needs and resources. This report provides detailed findings of this needs assessment and recommends specific activities related to outreach education, research and capacity building.

Kim, Y. (2012). Parenting needs for parents of young children in southern Nevada. University of Nevada Cooperative Extension. Special Publication, SP-12-10 . 

This study describes a needs assessment for parents of 5-year-old or younger children in southern Nevada. The needs assessment was carried out to identify parenting educational needs, given the relevance of children’s early years on their development and the limited resources available, especially for vulnerable families. The multi-method assessment included: a review of most critical areas in young children’s development, examination of well-being statistics, interviews with personnel working with young children and their families, and information collected from parents of young children about education needs. The study identified six needs to prioritize related to child learning, parents’ education, early literacy, self-discipline, healthy eating habits and community resources. The study also recommends that parenting information should be delivered through parents’ preferred mediums, and that more parenting education workshops and collaboration with other agencies in diverse geographical areas should occur.

Kratsch, H., & Skelly, J. (2012). Situational analysis: Horticultural needs and trends in Nevada. University of Nevada Cooperative Extension. Special Publication, SP-12-12.  

This report presents the results of needs assessments, studies, stakeholder interviews and surveys that were compiled to identify potential horticulture education targets and program efforts in Nevada. The report investigates trends in Nevada demographics, climate, horticulture, consumer horticulture, horticulture industry, Nevada green industry, and horticultural needs at the county level. Upon review of these topics, the report proposes a statewide emphasis on the following horticultural issues: home and small-scale food production, support of community – and school – garden program efforts, pest diagnosis and management, climate-appropriate gardening, and programs to support green-industry training.

Marshall, M., Bush, D., & Hayes, K. (2005). Extension programming for food entrepreneurs: An Indiana needs assessment. Journal of Extension, 43(5), 5RIB9 . 

This study describes a needs assessment for food entrepreneurs in Indiana. Purdue University Cooperative Extension specialists were designing a program to help food entrepreneurs develop value-added food products. This needs assessment was carried out to understand the needs of such food entrepreneurs and plan what services and information would be useful for Extension outreach. Extensionists from 86 counties were surveyed on topics of marketing, new business start-up, food regulations and food safety. The results of the study were used to develop a statewide workshop for food entrepreneurs.

Skelly, J., Singletary, L., Angle, J., Sepúlveda-Pulvirenti, E., & Moffitt, M. (2010). Addressing the needs of Nevada’s growing Latino population. University of Nevada Cooperative Extension. Special Publication, SP-10-08.  

This report documents that the Latino population in Nevada began growing in the 1960s, both in numbers and diversity. Nevada is now one of the top 10 states in the nation in the percentage of Latino population. Although Latinos are economically active, they are underrepresented in professional and technical positions. A large proportion of families did not have health insurance. This needs assessment is the first comprehensive effort to assess the Latino population in Nevada. Researchers reviewed extant literature, contacted numerous organizations interested in Latino issues, and developed a comprehensive list of issues that might interest Latinos. Results indicated that many of the issues participants identified as important to them were outside the purview of Extension, but may be addressed by other organizations. The results of this assessment support the need to develop Extension programs that use inclusive curriculum and trainings, and address issues identified within the Extension mission.

Smith, M., Meehan, C., & Dasher, H. S. (2009). Assessing volunteers' needs and interests to inform curriculum development in 4-H. Journal of Extension, 47(1), 1IAW3 . 

This paper describes a needs assessment of volunteers in 4-H animal and veterinary science. The needs assessment followed a mixed-methods design, which included focus groups and interviews. The results revealed content areas where volunteers needed the most resources and support. Volunteers also expressed their interest in educational activities to learn about the identified concepts. The results of the study helped UC Davis researchers develop two curricula on animal care and on biology, physiology and veterinary care to be used in youth development programs.

Walker, S. (2003). Building a state child care initiative: Applying principles of teamwork and collaboration. Journal of Extension, 41(3), 3FEA2 . 

This study is a description of a collaboration in Maryland to provide a coordinated childcare training and build a system of childcare service at the state level. The author describes the five-year coordination process, which consisted of statewide communication, identification of faculty needs, organization of coordinated conferences, developing a curriculum for childcare provider training and its implementation. Based on their experience, the author provides a list of recommendations for the effective development of a statewide team effort, which includes building on the good things that are present (capacities, skills and assets of people involved), empowering and rewarding individuals, ensuring trusted leadership, being responsive to the needs, and seeking resources.

• Altschuld, J. W., & Watkins, R. (2014). A primer on needs assessment: More than 40 years of research and practice. In J. W. Altschuld & R. Watkins (Eds.), Needs assessment: Trends and a view toward the future. New Directions for Evaluation, 144, 5–18.
• Angima, S., Etuk, L., & King, D. (2014). Using needs assessment as a tool to strengthen funding proposals. Journal of Extension, 52(6), TOT v52-6tt1. 
• Berkowitz, B., & Wadud, E. (2013). Identifying community assets and resources. In: The Community Tool Box, chapter 3, section 8. University of Kansas. 
• Caravella, J. (2006). A needs assessment method for extension educators. Journal of Extension, 44(1), 1TOT2. 
• Etling, A., & Thomas, M. (1995). Needs assessment for extension agents and other nonformal educators. Pennsylvania State Univ., University Park. Cooperative Extension Service. ED 388 774.
• Garst, B., & McCawley, P. (2015). Solving problems, ensuring relevance, and facilitating change: the evolution of needs assessment within Cooperative Extension. Journal of Human Sciences and Extension, 3(2), 26-47. 
• Ghimire, N. R. (2010). The relative importance of selected educational process professional competencies to extension educators in the North Central Region of USA. Doctoral Dissertation, Iowa State University, Ames.
• Jacobs, F. (1988). The five-tiered approach to evaluation: Context and implementation. In: H. B. Weiss & F. H. Jacobs (Eds), Evaluating family programs (pp 37-68). Hawthorne, NY: Aldine de Gruyter.Koundinya, V. (2010). An analysis of the food safety educational processes in the Cooperative Extension System of the North Central Region of the United States. Doctoral Dissertation, Iowa State University, Ames.
• Mathie, A., & Cunningham, G. (2003). From clients to citizens: Asset-based community development as a strategy for community-driven development. Development in Practice 13(5):474-486. 

Learn more about the author(s)

Also of Interest:

Batsche, C., Hernandez, M., & Montenegro, M. C. (1999). Community needs assessment with Hispanic, Spanish-monolingual residents. Evaluation and Program Planning, 22(1), 13-20 .

An EEO/AA Institution. Copyright © 2024 , University of Nevada Cooperative Extension. A partnership of Nevada counties; University of Nevada, Reno; and the U.S. Department of Agriculture

• Open access
• Published: 21 December 2015

Tools and instruments for needs assessment, monitoring and evaluation of health research capacity development activities at the individual and organizational level: a systematic review

• Johanna Huber 1 ,
• Sushil Nepal 1 ,
• Daniel Bauer 1 ,
• Insa Wessels 2 ,
• Martin R Fischer 1 &
• Claudia Kiessling 1 , 3  

Health Research Policy and Systems volume  13 , Article number:  80 ( 2015 ) Cite this article

28k Accesses

24 Citations

1 Altmetric

Metrics details

In the past decades, various frameworks, methods, indicators, and tools have been developed to assess the needs as well as to monitor and evaluate (needs assessment, monitoring and evaluation; “NaME”) health research capacity development (HRCD) activities. This systematic review gives an overview on NaME activities at the individual and organizational level in the past 10 years with a specific focus on methods, tools and instruments. Insight from this review might support researchers and stakeholders in systemizing future efforts in the HRCD field.

A systematic literature search was conducted in PubMed and Google Scholar. Additionally, the personal bibliographies of the authors were scanned. Two researchers independently reviewed the identified abstracts for inclusion according to previously defined eligibility criteria. The included articles were analysed with a focus on both different HRCD activities as well as NaME efforts.

Initially, the search revealed 700 records in PubMed, two additional records in Google Scholar, and 10 abstracts from the personal bibliographies of the authors. Finally, 42 studies were included and analysed in depth. Findings show that the NaME efforts in the field of HRCD are as complex and manifold as the concept of HRCD itself. NaME is predominately focused on outcome evaluation and mainly refers to the individual and team levels.

A substantial need for a coherent and transparent taxonomy of HRCD activities to maximize the benefits of future studies in the field was identified. A coherent overview of the tools used to monitor and evaluate HRCD activities is provided to inform further research in the field.

Peer Review reports

The capacity to cope with new and ill-structured situations is a crucial ability in today’s world. Developing this ability, by shaping empowered citizens, challenges individuals as well as organisations and societies. This process of empowerment is usually referred to as capacity development (CD) [ 1 ]. While this term has been commonly used for years in the field of foreign aid, other societal and political domains (e.g. social work, education and health systems) are increasingly adopting the concept of CD when developing new or existing competencies, structures, and strategies for building resilient individuals and organizations [ 2 ]. Also in the field of health research, an increasing number of activities to strengthen health research competencies and to support organizations can be observed – as demanded by the three United Nations Millennium Development Goals addressing health related issues [ 3 – 6 ]. Several frameworks are already in use that support a structured approach to health research capacity development (HRCD) and address competencies that are specific to health research [ 7 – 9 ]. These frameworks usually incorporate the individual or team, organization or institution, and society levels [ 8 , 10 , 11 ]. One conclusion that can be drawn from the available evidence is that, in such a structured approach to HRCD efforts, meaningful data collection is crucial. First, data collection incorporates the HRCD needs assessment and second, the monitoring and evaluation (NaME) of activities and programs once implemented. Therefore, HRCD activities should address the needs as assessed. Monitoring and evaluation of these activities should reflect the desired outcomes as defined beforehand [ 12 – 15 ]. Bates et al. [ 16 ] indicate how data collection tools and instruments are usually developed for a certain purpose in a certain context. The context specificity of tools and instruments has to be considered and the appropriateness of these must be determined when selecting instruments for any needs assessment for a new project. This article offers a systematic review of tools and instruments for the NaME of HRCD activities at the individual or team and the organizational levels to aid HRCD initiatives in selecting appropriate tools and instruments for data collection within their respective context. For this purpose, a range of studies published between January 1, 2003, and June 30, 2013, were chosen and analysed based on different context parameters such as the level of the CD and the nature of the HRCD activities.

We followed the PRISMA checklist for reporting systematic reviews and meta-analyses [ 17 ]. Inclusion and analysis criteria were defined in advance and documented in a protocol (Tables  1 and 2 ).

Information sources and search strategy

We conducted the systematic literature search in July 2013. The search was done in both the literature database PubMed and the search engine Google Scholar. We applied the three search terms “capacity building” AND “research ” , “capacity development” AND “research”, and “capacity strengthening” AND “research”. We checked the first 200 hits in Google Scholar for each search term. “Health” and “evaluation” were not included in the search terms as a pre-test search had revealed this would exclude relevant literature. Articles from personal bibliographies of the authors were also included.

Inclusion categories and criteria

The inclusion process was structured along the five inclusion categories ‘capacity development’, ‘research’, ‘health profession fields’, ‘monitoring and evaluation’, and ‘level of NaME’. Table  1 gives a detailed overview of all descriptions and operationalisations used.

The category ‘capacity development’ [ 18 ] represents an exemplary definition which serves as a guideline for inclusion but should not to be applied word by word. ‘Research’ was operationalized according to the categories of the ‘research spider’ [ 19 ]. Some process-related research skills as well as communicational and interpersonal skills were added to our operationalisation [ 20 ]. Main health professions were identified and grouped within different fields. NaME was operationalized according to a self-constructed NaME framework of HRCD activities (Fig.  1 ), which summarizes 13 HRCD/NaME frameworks [ 2 , 5 , 8 , 10 – 13 , 15 , 21 – 25 ] and reflects the level of HRCD, common indicators, and the order (from needs assessment to impact evaluation) commonly used in the original frameworks.

Framework for needs assessment, monitoring and evaluation (NaME) of health research capacity development (HRCD) [ 2 , 5 , 8 , 10 – 13 , 15 , 21 – 25 ].

For the categories ‘research’, ‘health profession fields’ and ‘monitoring and evaluation’, at least one of the operationalisations of each category had to be addressed by the study. The category ‘level of NaME’ was operationalized referring to the ESSENCE framework ‘Planning, monitoring and evaluation framework for capacity strengthening in health research’ which describes three CD levels: individual and/or team, organizational, and system levels [ 10 ]. Only publications focussing on NaME on the individual/team and organizational levels were considered for this review.

Additionally, the following eligibility criteria were set: English or German language, publication period from January 1, 2003, to June 30, 2013, intervention, non-intervention and multiple design studies (Fig.  2 ). We excluded grey literature, editorials, comments, congress abstracts, letters, and similar. Articles focussing on institutional networks with external partners were excluded as well.

Categorization of the study designs. The study designs are restricted to the included studies.

Study selection

Two researchers, JH and SN, independently scanned the abstracts identified for inclusion. In case of disagreement, JH and SN discussed the abstracts in question. If consensus could still not be reached, a third reviewer, CK, was consulted. After consensus on inclusion was reached, the full-texts of all included studies were rechecked for inclusion by JH and SN.

Study analysis procedure

We analysed the included articles according to nine aspects defined in Table  2 .

The search in PubMed revealed 700 suitable records (Fig.  3 ). We removed 27 duplicates, resulting in 673 records for inclusion screening. The first 200 hits for each of the three search terms in Google Scholar were considered, resulting in two additional records after removing duplicates. Furthermore, we included articles from the personal bibliographies of the authors, adding 10 more abstracts after checking for duplicates. Of the 685 records identified, 24 did not contain an abstract, but were preliminarily included for the full-text screening. JH and SN scanned the remaining 661 abstracts in terms of the inclusion criteria, thus excluding 616 records; 45 abstracts and the 24 records without abstracts were considered for full-text screening. After the full-text screening, 42 articles were finally included for further analysis; 37 articles originated from PubMed, one from Google Scholar, and four from the personal bibliographies of the authors.

Flowchart of the inclusion process.

These 42 articles were subsequently analysed along nine aspects (Table  2 ). The results are summarized in Table  3 .

Around half of the NaME studies on HRCD activities were conducted in high-income countries (n = 24) [ 26 ]. Six studies took place in lower-middle-income and two in upper-middle-income economies. Participants of one study were from a low-income country [ 27 ]. Two studies were performed in partnerships between a high-income and several low-, lower-middle and upper-middle-income economies. Mayhew et al. [ 28 ] described a partnership study between two upper-middle income countries and Bates et al. [ 29 ] analysed case studies from two lower-middle-income and two low-income economies. Five authors did not specify the country or region of their studies.

The evaluation focus of the studies was predominately on outcome evaluation (n = 23). Besides that, six studies surveyed the current state, three studies assessed requirements, and two studies investigated needs of HRCD activities. The remaining eight studies combined two evaluation aspects: definition of needs and outcome evaluation (n = 4), analysis of current state and outcome evaluation (n = 1), outcome evaluation and impact evaluation (n = 1), and analysis of current state and definition of needs (n = 1). Jamerson et al. [ 30 ] did not define their focus of evaluation.

Nearly half of the studies investigated HRCD on the individual/team level (n = 20); 16 studies were conducted at both the individual/team and organizational levels. The authors of six studies focused on organizational aspects of HRCD.

Almost all studies (n = 38) described and evaluated HRCD activities; 19 of these HRCD activities were training programmes of predefined duration, lasting between some hours or days up to 2 years. Another nine HRCD activities were perpetual or their duration not specified and 10 studies defined and pre-assessed the setting in preparation of an HRCD activity. The authors of four studies did not specify an HRCD activity, focussing on the development or validation of tools, instruments, and frameworks.

The participants of HRCD activities represent a wide range of health professions (e.g. laboratory scientists, physiotherapists, dentists, pharmacists); 10 studies investigated staff with management tasks in health, e.g. hospital managers, clinical research managers. Nurses participated in eight studies with another eight studies looking into ‘research staff’ and ‘scientists’ with no further description. Medical practitioners were studied in five papers. Besides all these, the background of participants was often not specified beyond general terms like ‘health professionals’, ‘ethic committee members’, ‘scholars’, ‘university faculty members’, or ‘allied health professionals’. In a different approach, Suter et al. [ 31 ] analysed reports and Bates et al. [ 29 ] investigated case studies (without specifying the material scrutinized).

A wide variety of study designs was employed by the studies included in the review. We identified 35 single-study and six multi-study approaches. Of the 35 single-study approaches, 10 were designed as intervention (three with control groups) and 25 as non-intervention studies. Four multi-study approaches combined an intervention study with a non-intervention study. Two multi-study approaches combined different non-intervention studies. Jamerson et al. [ 30 ] did not specify their study design.

Many different tools and instruments for NaME were identified and applied in quantitative, qualitative and mixed mode of analysis. No preferred approach was observed. One third of the studies (n = 16) used a combination of tools for quantitative as well as qualitative analysis. In 13 studies, tools like questionnaires and assessment sheets were applied to evaluate and monitor HRCD activities quantitatively. Evaluation tools, such as interviews, focus group discussions, document analyses, or mapping of cases against evaluation frameworks, were identified in 12 studies and commonly analysed in a qualitative approach. In one study, tools for evaluation were not described at all.

Summary of evidence

The aim of our systematic review was to give an overview on tools and instruments for NaME of HRCD activities on the individual and organizational level; 42 included articles demonstrated a large variety of tools and instruments in specific settings. Questionnaires, assessment sheets and interviews (in qualitative settings) were most commonly applied and in part disseminated for further use, development and validation.

Overall, 36 studies were either conducted on the individual/team or on both individual/team and organizational level. Within these studies, a well-balanced mixture of quantitative, qualitative and mixed tools and modes of analysis were applied. Judging from the depth of these studies, it seems as if NaME of HRCD on the individual level is quite well developed. Only six studies focused exclusively on organizational aspects, almost all with qualitative approaches, indicating that HRCD studies at this level are still mainly exploratory. The organizational level is possibly a more complex construct to measure. The fact that 13 out of 19 studies that broach organizational aspects were conducted in high-income countries might reflect the wider possibilities of these research institutions and indicates a need for more attention to NaME on the organizational level in lower-income settings. Results from these exploratory studies on the organizational level should feed into the development of standardized quantitative indicators more regularly. Qualitative approaches could be pursued for complex and specific constructs not easily covered quantitatively.

By not limiting the primary selection of articles for this review to a specific health profession, it was revealed that staff with management tasks in health research, as well as nurses, were the cohorts most frequently targeted by NaME studies. Further research should concentrate on other health professionals to determine communalities and differences of health-research related skill acquisition and development between health professions. These studies could determine whether and which parts of HRCD and NaME can be considered generic across health professions. Further, we will at some point have to ask, who is being left out and who is not getting access to HRCD programs, and why.

The focus of NaME throughout the studies included in this review was on outcome measurement, regardless of whether these were conducted in high-income, upper-middle, lower-middle, or low-income countries. However, there were only few reports of needs assessment from middle- and low-income economies, while high-income countries regularly give account of current states. While this should not be over-interpreted, it still raises the question of whether the needs assessment in the middle- and low-income countries is being done as thoroughly as warranted, but not reported in the articles, or if these countries’ needs might not always be at the very centre of the HRCD’s attention. While the evaluation of HRCD outcomes is, of course, of importance, more attention should be paid to the sustainability of programs and impact evaluation, e.g. parameters of patient care or societal aspects. Only one study, that of Hyder et al. [ 32 ], made use of one such indicator and assessed the impact of a HRCD training by considering “teaching activities after returning to Pakistan”. The development of valid impact indicators of course constitutes a methodological challenge. Some studies reporting impact evaluation on a system level might of course have been missed due to the search parameters applied.

When undertaking the review, three main methodological weaknesses of this research area became apparent. First, there is a need for common definitions and terminologies to better communicate and compare the HRCD efforts. The analysis of the studies showed that there is an inconsistent use of terms, for example, for CD activities (e.g. training, course, or workshop). Similar problems were already identified in the context of educational capacity building by Steinert et al. [ 33 ], who suggest definitions for different training settings which may also be suitable for a more precise description of CD activities. A common taxonomy for the description of health professionals (i.e. the study participants) would be just as desirable. The use of coherent terms would not only enable the accurate replication of studies but also help in determining whether tools and instruments from one setting can be easily transferred to another. A clear and coherent description of study setting and participants is thus an integral step towards scientific transparency. The incoherent categorisation of study types is probably not a new problem. It is, however, amplified by authors who choose very complex approaches to collect data at different NaME levels with deviating terms to describe these approaches [ 28 , 34 – 36 ].

The second weakness of the research area is the varying adherence to reporting standards. While there are standards available for reporting qualitative or quantitative research (e.g. Rossi et al. [ 12 ], Downing [ 37 ], Mays & Pope [ 38 ]), it seems these or similar recommendations were not frequently considered when reporting or reviewing NaME studies. This was particularly the case in studies with a mixed-method mode of analysis, where the need for more standardised reporting became apparent. Frambach et al.’s [ 39 ] “Quality Criteria in Qualitative and Quantitative Research” could provide guidance, especially for studies with mixed-method approaches. Another important aspect of transparent reporting would be the publication of the tools and instruments used in NaME studies. Of the 42 articles scrutinized during this review, only 15 either disclosed the tools and instruments within the article itself in an appendix or volunteered to have them sent to any audience interested. Of all the tools and instruments disclosed, only two were used in two or more studies. Making the tools and instruments available to the HRCD community would not only allow for their adaptation whenever necessary but, more importantly, support their validation and enhancement.

The last point concerns the study designs implemented. The majority of articles are mainly descriptive, non-intervention studies that only allow for low evidence according to Cochrane standards [ 40 ]. While most HRCD studies conducted in high-income economies were of non-interventional nature, those from low- and middle-income countries were a mix of non-intervention, intervention and multi-study approaches, yielding higher levels of evidence. Of all interventional studies, most employed a quasi-experimental design with only one randomized controlled trial [ 23 ]. The studies reporting HRCD on the institutional level were also primarily on a descriptive level. Cook et al. [ 41 ], however, demand going beyond describing what one did (descriptive studies) or whether an intervention worked or not (justification studies). Instead, they call for analysing how and why a program worked or failed (clarification studies). An in-depth analysis of the effectiveness of different HRCD activities is, however, still lacking.

Limitations of the systematic review

This systematic review displays some methodological limitations itself. The issue of deviating terminologies has been raised earlier. In most cases, we adopted the terms used in the studies themselves, e.g. when reporting the authors’ denoted study designs. In very few cases, we changed or completed terms to make the studies more comparable to others. One example is changing the wording from Green et al.’s [ 35 ] “case study approach” into a “multi-study approach” to match Flyvberg’s taxonomy [ 42 ]. Other limitations typical for reviews may also apply. Relevant sources might not have been detected due to the selected search terms, the range of the data sources, the exclusion of grey literature, and the restriction to English and German sources.

A systematic review on studies from the field of HRCD activities was conducted, with 42 studies being fully analysed. The analysis revealed that a variety of terms and definitions used to describe NaME efforts impedes the comparability and transferability of results. Nevertheless, insight from this review can help to inform researchers and other stakeholders in the HRCD community. A coherent overview on tools and instruments for NaME of HRCD was developed and is provided (Table  3 ).

Furthermore, it is time to set standards for NaME in the HRCD community. Researchers and stakeholders should develop a common research agenda to push, systematise and improve the research efforts in the field of NaME of HRCD activities. To do so, a common language and terminology is required. The conceptualizations used for the purpose of these review can inform this development. On the other hand, we have to critically analyse research gaps in terms of generalizable versus context-specific theories, methods, tools, and instruments. To maximize the benefits and to incorporate different research traditions, these undertakings should be done internationally and multi-professionally within the HRCD community.

Lusthaus C, Adrien M-H, Perstinger M. Capacity Development: Definitions, Issues and Implications for Planning, Monitoring and Evaluation. Universalia Occasional Paper. 1999;35:[about 21 p.]. http://preval.org/documentos/2034.pdf . Accessed 2 July 2015.

Labin SN, Duffy JL, Meyers DC, Wandersman A, Lesesne CA. A research synthesis of the evaluation capacity building literature. Am J Eval. 2012;33(3):307–38.

Article   Google Scholar  

Gadsby EW. Research capacity strengthening: donor approaches to improving and assessing its impact in low- and middle-income countries. Int J Health Plann Manage. 2011;26(1):89–106. doi: 10.1002/hpm.1031 .

Article   PubMed   Google Scholar  

Bailey J, Veitch C, Crossland L, Preston R. Developing research capacity building for Aboriginal & Torres Strait Islander health workers in health service settings. Rural Remote Health. 2006;6(4):556.

PubMed   CAS   Google Scholar  

Lansang MA, Dennis R. Building capacity in health research in the developing world. Bull World Health Organ. 2004;82(10):764–70.

PubMed   PubMed Central   Google Scholar  

United Nations. Millennium Development Goals and Beyond. 2015. http://www.un.org/millenniumgoals/ . Accessed 2 July 2015.

Google Scholar  

Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst. 2011;9:26. doi: 10.1186/1478-4505-9-26 .

Article   PubMed   PubMed Central   Google Scholar  

Cooke J. A framework to evaluate research capacity building in health care. BMC Fam Pract. 2005;6:44. doi: 10.1186/1471-2296-6-44 .

Trostle J. Research capacity building in international health: definitions, evaluations and strategies for success. Soc Sci Med. 1992;35(11):1321–4.

Article   PubMed   CAS   Google Scholar  

ESSENCE (Enhancing Support for Strengthening the Effectiveness of National Capacity Efforts). Planning, monitoring and evaluation framework for capacity strengthening in health research. Geneva: ESSENCE Good practice document series; 2011. http://whqlibdoc.who.int/hq/2011/TDR_essence_11.1_eng.pdf?ua = 1. Accessed 2 July 2015.

Ghaffar A, IJsselmuiden C, Zicker F. Changing mindsets: Research capacity strengthening in low- and middle-income countries. Geneva: Council on Health Research for Development (COHRED), Global Forum for Health Research, INICEF/UNDP/World Bank/WHO Special Programme for Research and Training in Tropical Diseases (TDR); 2008. http://www.cohred.org/downloads/cohred_publications/Changing_Mindsets.pdf . Accessed 30 November 2015.

Rossi PH, Lipsey MW, Freeman HE. Evaluation. A systematic approach. 7th ed. Thousand Oaks: Sage Publications; 2004.

Donabedian A. The quality of care. How can it be assessed? 1988. Arch Pathol Lab Med. 1997;121(11):1145–50.

McLaughlin JA, Jordan GB. Logic models: a tool for telling your programs performance story. Eval Program Plan. 1999;22(1):65–72.

Best A, Terpstra JL, Moor G, Riley B, Norman CD, Glasgow RE. Building knowledge integration systems for evidence-informed decisions. J Health Organ Manage. 2009;23(6):627–41.

Bates I, Boyd A, Smith H, Cole DC. A practical and systematic approach to organisational capacity strengthening for research in the health sector in Africa. Health Res Policy Syst. 2014;12:11. doi: 10.1186/1478-4505-12-11 .

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100. doi: 10.1371/journal.pmed.1000100 . Epub 2009 Jul 21.

Deutsche Gesellschaft für Technische Zusammenarbeit (GTZ) GmbH. Capacity WORKS: The Management Model for Sustainable Development. Eschborn: GTZ; 2009.

Smith H, Wright D, Morgan S, Dunleavey J. The ‘Research Spider’: a simple method of assessing research experience. Prim Health Care Res Dev. 2002;3:139–40.

Bauer D, Wessels I, Huber J, Fischer MR, Kiessling C. Current State and Needs Assessment for Individual and Organisational Research Capacity Strengthening in Africa: A Case Report from Mbeya (TZ), and Kumasi (GH). Eschborn: Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH; 2013.

Bates I, Akoto AY, Ansong D, Karikari P, Bedu-Addo G, Critchley J, et al. Evaluating health research capacity building: An evidence-based tool. PLoS Med. 2006;3(8), e299.

Golenko X, Pager S, Holden L. A thematic analysis of the role of the organisation in building allied health research capacity: a senior managers’ perspective. BMC Health Serv Res. 2012;12:276. doi: 10.1186/1472-6963-12-276 .

Ijsselmuiden C, Marais DL, Becerra-Posada F, Ghannem H. Africa’s neglected area of human resources for health research - the way forward. S Afr Med J. 2012;102(4):228–33.

Levine R, Russ-Eft D, Burling A, Stephens J, Downey J. Evaluating health services research capacity building programs: Implications for health services and human resource development. Eval Program Plan. 2013;37:1–11. doi: 10.1016/j.evalprogplan.2012.12.002 . Epub 2012 Dec 12.

Mahmood S, Hort K, Ahmed S, Salam M, Cravioto A. Strategies for capacity building for health research in Bangladesh: Role of core funding and a common monitoring and evaluation framework. Health Res Policy Syst. 2011;9:31. doi: 10.1186/1478-4505-9-31 .

The World Bank. Data: Country and Lending Groups. 2015. http://data.worldbank.org/about/country-and-lending-groups . Accessed 2 July 2015.

Njie-Carr V, Kalengé S, Kelley J, Wilson A, Muliira JK, Nabirye RC, et al. Research capacity-building program for clinicians and staff at a community-based HIV clinic in Uganda: a pre/post evaluation. J Assoc Nurses AIDS Care. 2012;23(5):431–41. doi: 10.1016/j.jana.2011.11.002 .

Mayhew SH, Doherty J, Pitayarangsarit S. Developing health systems research capacities through north–south partnership: an evaluation of collaboration with South Africa and Thailand. Health Res Policy Syst. 2008;6:8. doi: 10.1186/1478-4505-6-8 .

Bates I, Taegtmeyer M, Squire BS, Ansong D, Nhlema-Simwaka B, Baba A, et al. Indicators of sustainable capacity building for health research: analysis of four African case studies. Health Res Policy Syst. 2011;9:14. doi: 10.1186/1478-4505-9-14 .

Jamerson PA, Fish AF, Frandsen G. Nursing Student Research Assistant Program: A strategy to enhance nursing research capacity building in a Magnet status pediatric hospital. Appl Nurs Res. 2011;24(2):110–3. doi: 10.1016/j.apnr.2009.08.004 .

Suter E, Lait J, Macdonald L, Wener P, Law R, Khalili H, et al. Strategic approach to building research capacity in inter-professional education and collaboration. Healthc Q. 2011;14(2):54–60.

Hyder AA, Akhter T, Qayyum A. Capacity development for health research in Pakistan: the effects of doctoral training. Health Policy Plan. 2003;18(3):338–43.

Steinert Y, Mann K, Centeno A, Dolmans D, Spencer J, Gelula M, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach. 2006;28(6):497–526.

Ajuwon AJ, Kass N. Outcome of a research ethics training workshop among clinicians and scientists in a Nigerian university. BMC Medical Ethics. 2008;9:1. doi: 10.1186/1472-6939-9-1 .

Green B, Segrott J, Hewitt J. Developing nursing and midwifery research capacity in a university department: case study. J Adv Nurs. 2006;56(3):302–13.

Janssen J, Hale L, Mirfin-Veitch B, Harland T. Building the research capacity of clinical physical therapists using a participatory action research approach. Phys Ther. 2013;93(7):923–34. doi: 10.2522/ptj.20120030 .

Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–7.

Mays N, Pope C. Qualitative research in health care. Assessing quality in qualitative research. BMJ. 2000;320(7226):50–2.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Frambach JM, van der Vleuten CP, Durning SJ. AM last page. Quality criteria in qualitative and quantitative research. Acad Med. 2013;88(4):552. doi: 10.1097/ACM.0b013e31828abf7f .

Cochrane Consumer Network. Levels of Evidence. 2015. http://consumers.cochrane.org/levels-evidence#about-cochrane . Accessed 30 November 2015.

Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of research in medical education. Med Educ. 2008;42(2):128–33. doi: 10.1111/j.1365-2923.2007.02974.x . Epub 2008 Jan 8.

Bent Flyvbjerg. Five misunderstandings about case-study research. Qual Inq. 2006;12:219. doi: 10.1177/1077800405284363 .

Ali J, Hyder AA, Kass NE. Research ethics capacity development in Africa: exploring a model for individual success. Dev World Bioeth. 2012;12(2):55–62. doi: 10.1111/j.1471-8847.2012.00331.x .

Barchi FH, Kasimatis-Singleton M, Kasule M, Khulumani P, Merz JF. Building research capacity in Botswana: a randomized trial comparing training methodologies in the Botswana ethics training initiative. BMC Med Educ. 2013;13:14. doi: 10.1186/1472-6920-13-14 .

Bates I, Ansong D, Bedu-Addo G, Agbenyega T, Akoto AY, Nsiah-Asare A, et al. Evaluation of a learner-designed course for teaching health research skills in Ghana. BMC Med Educ. 2007;7:18.

Bullock A, Morris ZS, Atwell C. Collaboration between health services managers and researchers: making a difference? J Health Serv Res Policy. 2012;17 Suppl 2:2–10. doi: 10.1258/jhsrp.2011.011099 .

Kirkpatrick DL. Evaluating training programs. Mumbai: Tata McGraw-Hill Education; 1998.

Kirkpatrick DL. Techniques for evaluating training programs. Training Dev J. 1979;33:78–92.

Cooke J, Nancarrow S, Dyas J, Williams M. An evaluation of the ‘Designated Research Team’ approach to building research capacity in primary care. BMC Fam Pract. 2008;9:37. doi: 10.1186/1471-2296-9-37 .

Corchon S, Portillo MC, Watson R, Saracíbar M. Nursing research capacity building in a Spanish hospital: an intervention study. J Clin Nurs. 2011;20(17–18):2479–89. doi: 10.1111/j.1365-2702.2011.03744.x .

Dodani S, La Porte RE. Ways to strengthen research capacity in developing countries: effectiveness of a research training workshop in Pakistan. Public Health. 2008;122(6):578–87. doi: 10.1016/j.puhe.2007.09.003 .

du Plessis E, Human SP. Reflecting on ‘meaningful research’: a qualitative secondary analysis. Curationis. 2009;32(3):72–9.

Finch E, Cornwell P, Ward EC, McPhail SM. Factors influencing research engagement: research interest, confidence and experience in an Australian speech-language pathology workforce. BMC Health Serv Res. 2013;13:144. doi: 10.1186/1472-6963-13-144 .

Holden L, Pager S, Golenko X, Ware RS, Weare R. Evaluating a team-based approach to research capacity building using a matched-pairs study design. BMC Fam Pract. 2012;13:16. doi: 10.1186/1471-2296-13-16 .

Henderson-Smart DJ, Lumbiganon P, Festin MR, Ho JJ, Mohammad H, McDonald SJ, et al. Optimising reproductive and child health outcomes by building evidence-based research and practice in South East Asia (SEA-ORCHID): study protocol. BMC Med Res Methodol. 2007;7:43.

Holden L, Pager S, Golenko X, Ware RS. Validation of the research capacity and culture (RCC) tool: measuring RCC at individual, team and organisation levels. Aust J Prim Health. 2012;18(1):62–7. doi: 10.1071/PY10081 .

Hyder AA, Harrison RA, Kass N, Maman S. A case study of research ethics capacity development in Africa. Acad Med. 2007;82(7):675–83.

Jones A, Burgess TA, Farmer EA, Fuller J, Stocks NP, Taylor JE, et al. Building research capacity. An exploratory model of GPs’ training needs and barriers to research involvement. Aust Fam Physician. 2003;32(11):957–60.

PubMed   Google Scholar  

Kwon S, Rideout C, Tseng W, Islam N, Cook WK, Ro M, et al. Developing the community empowered research training program: building research capacity for community-initiated and community-driven research. Prog Community Health Partnersh. 2012;6(1):43–52. doi: 10.1353/cpr.2012.0010 .

Lazzarini PA, Geraghty J, Kinnear EM, Butterworth M, Ward D. Research capacity and culture in podiatry: early observations within Queensland Health. J Foot Ankle Res. 2013;6(1):1. doi: 10.1186/1757-1146-6-1 .

McIntyre E, Brun L, Cameron H. Researcher development program of the primary health care research, evaluation and development strategy. Aust J Prim Health. 2011;17(1):114–21. doi: 10.1071/PY10049 .

Minja H, Nsanzabana C, Maure C, Hoffmann A, Rumisha S, Ogundahunsi O, et al. Impact of health research capacity strengthening in low- and middle-income countries: the case of WHO/TDR programmes. PLoS Negl Trop Dis. 2011;5(10):e1351. doi: 10.1371/journal.pntd.0001351 .

Moore J, Crozier K, Kite K. An action research approach for developing research and innovation in nursing and midwifery practice: building research capacity in one NHS foundation trust. Nurse Educ Today. 2012;32(1):39–45. doi: 10.1016/j.nedt.2011.01.014 .

Otiniano AD, Carroll-Scott A, Toy P, Wallace SP. Supporting Latino communities’ natural helpers: a case study of promotoras in a research capacity building course. J Immigr Minor Health. 2012;14(4):657–63. doi: 10.1007/s10903-011-9519-9 .

Pager S, Holden L, Golenko X. Motivators, enablers, and barriers to building allied health research capacity. J Multidiscip Healthc. 2012;5:53–9. doi: 10.2147/JMDH.S27638 .

Perry L, Grange A, Heyman B, Noble P. Stakeholders’ perceptions of a research capacity development project for nurses, midwives and allied health professionals. J Nurs Manag. 2008;16(3):315–26. doi: 10.1111/j.1365-2834.2007.00801.x .

Priest H, Segrott J, Green B, Rout A. Harnessing collaboration to build nursing research capacity: a research team journey. Nurse Educ Today. 2007;27(6):577–87.

Green B, Segrott J, Coleman M, Cooke A. Building the research capacity of an academic department of nursing. Occasional Paper 1a. Swansea: School of Health Sciences, University of Wales; 2005.

Segrott J, Green B, McIvor M. Challenges and strategies in developing nursing research capacity: a review of the literature. Int J Nurs Stud. 2006;43(5):637–51.

Redman-MacLaren M, MacLaren DJ, Harrington H, Asugeni R, Timothy-Harrington R, Kekeubata E, et al. Mutual research capacity strengthening: a qualitative study of two-way partnerships in public health research. Int J Equity Health. 2012;11:79. doi: 10.1186/1475-9276-11-79 .

Ried K, Farmer EA, Weston KM. Setting directions for capacity building in primary health care: a survey of a research network. BMC Fam Pract. 2006;7:8.

Salway S, Piercy H, Chowbey P, Brewins L, Dhoot P. Improving capacity in ethnicity and health research: report of a tailored programme for NHS Public Health practitioners. Prim Health Care Res Dev. 2013;14(4):330–40. doi: 10.1017/S1463423612000357 .

Webster E, Thomas M, Ong N, Cutler L. Rural research capacity building program: capacity building outcomes. Aust J Prim Health. 2011;17(1):107–13. doi: 10.1071/PY10060 .

Wilson LL, Rice M, Jones CT, Joiner C, LaBorde J, McCall K, et al. Enhancing research capacity for global health: evaluation of a distance-based program for international study coordinators. J Contin Educ Health Prof. 2013;33(1):67–75. doi: 10.1002/chp.21167 .

Wootton R. A simple, generalizable method for measuring individual research productivity and its use in the long-term analysis of departmental performance, including between-country comparisons. Health Res Policy Syst. 2013;11:2. doi: 10.1186/1478-4505-11-2 .

Download references

Author information

Authors and affiliations.

Institut für Didaktik und Ausbildungsforschung in der Medizin, Klinikum der Universität München, Ziemssenstraße 1, 80336, Munich, Germany

Johanna Huber, Sushil Nepal, Daniel Bauer, Martin R Fischer & Claudia Kiessling

bologna.lab, Humboldt-Universität zu Berlin, Hausvogteiplatz 5-7, 10117, Berlin, Germany

Insa Wessels

Medizinische Hochschule Brandenburg Theodor Fontane, Fehrbelliner Straße 38, 16816, Neuruppin, Germany

Claudia Kiessling

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Johanna Huber .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

JH and SN designed and conducted the systematic review. JH wrote the draft of the systematic review and revised it according to the commentaries of SN, DB, IW, MF, and CK. JH provided the final version of the manuscript. SN additionally critically reviewed the manuscript and substantially contributed to the final version of the manuscript. DB critically reviewed both the design of the systematic review as well as the manuscript. He was involved in the development of meaningful inclusion criteria. DB contributed substantially to the final version of the manuscript. IW critically reviewed the design of the study and made important suggestions for improvement. She also critically reviewed the manuscript and contributed substantially to the final version of the manuscript. MF critically reviewed the design of the study and the manuscript. He suggested important improvements for the design of the study and substantially contributed to the final version of the manuscript. CK made substantial contributions to the design, conduction and review of the study, and was the third reviewer during the inclusion process of the identified studies. She critically reviewed the manuscript and delivered important improvements for the final version of the manuscript.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Huber, J., Nepal, S., Bauer, D. et al. Tools and instruments for needs assessment, monitoring and evaluation of health research capacity development activities at the individual and organizational level: a systematic review. Health Res Policy Sys 13 , 80 (2015). https://doi.org/10.1186/s12961-015-0070-3

Download citation

Received : 23 July 2015

Accepted : 07 December 2015

Published : 21 December 2015

DOI : https://doi.org/10.1186/s12961-015-0070-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health research capacity development
• Individual level
• Monitoring and evaluation
• Needs assessment
• Organizational level

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Open access
• Published: 17 January 2023

A scoping review of community health needs and assets assessment: concepts, rationale, tools and uses

• Hamid Ravaghi 1 ,
• Ann-Lise Guisset 2 ,
• Samar Elfeky 3 ,
• Naima Nasir 4 ,
• Sedigheh Khani 5 ,
• Elham Ahmadnezhad 6 &
• Zhaleh Abdi 7  

BMC Health Services Research volume  23 , Article number:  44 ( 2023 ) Cite this article

24k Accesses

16 Citations

2 Altmetric

Metrics details

Community health needs and assets assessment is a means of identifying and describing community health needs and resources, serving as a mechanism to gain the necessary information to make informed choices about community health. The current review of the literature was performed in order to shed more light on concepts, rationale, tools and uses of community health needs and assets assessment.

We conducted a scoping review of the literature published in English using PubMed, Embase, Scopus, Web of Science, PDQ evidence, NIH database, Cochrane library, CDC library, Trip, and Global Health Library databases until March 2021.

A total of 169 articles including both empirical papers and theoretical and conceptual work were ultimately retained for analysis. Relevant concepts were examined guided by a conceptual framework. The empirical papers were dominantly conducted in the  United States. Qualitative, quantitative and mixed-method approaches were used to collect data on community health needs and assets, with an increasing trend of using mixed-method approaches. Almost half of the included empirical studies used participatory approaches to incorporate community inputs into the process.

Our findings highlight the need for having holistic approaches to assess community’s health needs focusing on physical, mental and social wellbeing, along with considering the broader systems factors and structural challenges to individual and population health. Furthermore, the findings emphasize assessing community health assets as an integral component of the process, beginning foremost with community capabilities and knowledge. There has been a trend toward using mixed-methods approaches to conduct the assessment in recent years that led to the inclusion of the voices of all community members, particularly vulnerable and disadvantaged groups. A notable gap in the existing literature is the lack of long-term or longitudinal–assessment of the community health needs assessment impacts.

Peer Review reports

The population-based health approach aims to improve the population’s health, promote community resilience and reduce health inequities across the socioeconomic gradient via inter-sectoral partnerships among community groups, government, healthcare systems, and other stakeholders [ 1 ]. One key feature for adopting a population-based health approach is to ensure that it is grounded on a solid understanding of community health needs and assets by triangulating evidence from service providers and community members on services availability, accessibility, utilization and experience [ 2 , 3 ]. The process of identification of unmet health needs in a population is crucial for local authorities seeking to plan appropriate and effective programmes to meet these needs [ 3 , 4 ]. If these needs are ignored, then there is a risk of a top-down approach for providing health services, reflecting what a few people perceive to be the needs of the population rather than what they actually are [ 4 , 5 ].

In this context, community health needs assessment is a means of developing a comprehensive understanding of a community’s health and health needs as well as designing interventions to improve community health [ 6 ]. Though the process of community health needs assessment can be conducted in several ways, the primary purpose is to provide community leaders or healthcare providers with an overview of local policy, systems, and environmental change strategies currently in place and help to identify areas for improvement [ 7 ]. Community health needs assessment can provide them with a more nuanced understanding of the communities they serve, making them aware of pressing issues that require system-level changes and support their efforts for resource mobilization to initiate innovative programmes [ 8 , 9 ]. The process to gather evidence on community health needs can also serve as a springboard to strengthen community engagement [ 10 ].

In general, needs assessments are usually designed to evaluate gaps between current situations and desired outcomes, along with possible solutions to address the gaps. Recently, there has been a trend to move away from framing a community with a deficit perspective (need-based approach) to focus on community assets and resources, called community health needs and assets assessment [ 11 , 12 ]. In contrast to a need-based perspective which focuses on local deficits and resources outside the community, an asset-based perspective focuses on honing and leveraging existing strengths within the community to address community needs [ 12 , 13 , 14 ].

Studies have shown that community health needs assessment is used widely by different users and across different settings [ 15 , 16 ]. However, these studies varied widely in terms of purpose, process and methods of conducting community health needs assessment. Furthermore, the extent to which an asset-based approach is used is unclear, beyond the inclusion in guidance and recommendations. Thus, to support national or local decision-makers to make informed choices about the scope, tools, methods and use of community health needs and assets assessment, this scoping review of the literature aimed at: 1) Providing conceptual clarity on community health needs and assets assessment, 2) Determining for what purpose and with what methods community health needs and assets assessment are used globally, 3) Drawing the lessons learnt from previous experience with community health needs and assets assessment: what works in what context and under what conditions, 4) Documenting evidence of impact of community health needs and assets assessment, 5) Consolidating tools and methods used to collect evidence/data underpinning community health needs and assets assessment processes.

Search strategy

Ten databases, including PubMed, Embase, Scopus, Web of Science, PDQ evidence, NIH database, Cochrane library, CDC library, Trip, and Global Health Library were searched in February and March 2021. The search strategy was developed through discussion with experts in the field of population health, a research librarian, and a narrative review of the literature. Preliminary search terms were developed by the research team to reflect a number of core concepts including needs, population, needs assessment, assets assessment and participation. The search process was performed by a librarian with expertise in the use of literature databases (SK). The search terms were pilot-tested and agreed upon within the research team. The PubMed database search strategy presented in Additional file  1 .

Inclusion and exclusion criteria

Studies that focus on community health needs and assets assessment in terms of concepts, rationale, uses and tools were considered in both high-income countries (HICs) and low-and middle-income counties (LIMCs). We included studies in the review if they met the following criteria: 1) Papers providing conceptual clarity and explaining rationale for community health needs and (assets) assessment (This can be articles describing community health needs assessment or community assets assessment or community health needs and assets assessments at the same time or separately). The terms capabilities/ strengths/ resources can be used in place of assets and were considered.); 2) Papers describing or evaluating experiences implementing community health needs (and assets) assessment in a single site or multiple sites; 3) Methodological papers describing tools/approaches for community health needs (and assets) assessment; 4) Review of the literature on community health needs (and assets) assessment.

Types of papers not include in the review were: 1) Studies without a clear description of the community health needs and (assets) assessment methods, 2) Studies assessed a single dimension (i.e. health outcomes only, or healthcare providers’ capabilities only such as patient surveys, health outcomes dashboard, health facility assessment), 3) Studies related to a single disease or programme, 4) Studies focused only on engaging individual patient in their own care, and 5) Studies were not in English.

Three reviewers participated in the selection of the relevant studies (HR, ZA, NN). The eligibility and relevance of the articles were determined by two reviewers independently using the above predefined criteria. In the event of disagreement, a consensus was found between all the reviewers about the status of the article.

Data extraction

Separate data extraction forms were developed for the extraction of the three main categories of papers: conceptual, empirical and review papers. Totally, 121 empirical papers (including 6 review papers) and 48 conceptual and methodological papers were reviewed. Following topics were extracted for empirical papers: 1) General characteristics including author(s), year of publication, country of implementation, study objective(s) and study method; 2) Community health needs and (assets) assessment framing including rational, definitions of community health needs and (assets) assessment/ needs/ assets/ community, initiator(s) or user(s) of the process; 3) Key steps of the process, collected data, data collection tools; 4) Community engagement and the level of engagement; 5) Use of community health needs and (assets) assessment findings, impact of community health needs and (assets) assessment; 6) Facilitators and barriers. Data extraction forms are presented in Additional file  2 .

Data extraction forms were pilot-tested prior to the implementation. Two authors (ZA, HR) independently performed a pilot data extraction of a random sample of ten original articles. After piloting, the authors assessed the extracted data in relation to the scoping review questions and revised them accordingly. The content of the form was finalized by discussion within the team. Regarding conceptual papers, two authors (NN and ZA) initially extracted data from three randomly selected papers and subsequently refined and amended the form having research team inputs.

Four reviewers extracted included studies independently. The data extracted were cross-checked by one of the authors and mutual consensus resolved discrepancies. Individual data extraction forms of empirical papers were then merged into a single, unifying document used for the interpretation and presentation of the results. Following typical scoping review methods, the methodological quality of the included articles was not assessed systematically, however, only peer-reviewed articles were included in our review process [ 17 ].

Synthesis of results

Following reading and extracting conceptual papers, a preliminary conceptual framework (Fig.  1 ) was developed and discussed and agreed upon by team members. The integrative synthesis of the evidence was employed. Specifically, it involved the narrative description of concepts and definitions, key steps of the community health needs assessment and barriers and facilitators of the implementing community health needs assessment.

Conceptual framework of the review

The study selection process is summarized in Fig.  2 . Just over 12,000 records were obtained from the ten databases searched. Articles with obviously irrelevant titles were excluded, as were news items, letters, editorials, book reviews, and articles appearing in newsletters or magazines rather than peer review journals. The remaining abstracts were retrieved, read and assessed. A total of 169 articles including both empirical papers and theoretical and conceptual work were ultimately retained for analysis. A list of all studies with a short description, including the year of publication, key focus, study period, and methods, is presented in Additional files  3 and 4 . The first part of the results section focuses on definitions and concepts of community health needs assessment using both conceptual and empirical papers. In the second part of the results section, we describe key steps of the community health needs assessment and tools and methods used to collect data through content analysis of 121 included empirical papers. We also report some important challenges and facilitators faced by included studies while performing community health needs assessment. Role of community participation in the process and the spectrum and types of the participation is discussed in the last part.

Information flow in scoping review

General characteristics of the included studies

The review showed that community health needs assessment is used widely by different users and across different settings in both HICs and LMICs. Among included empirical studies, 81 (out of 121) were conducted in the  United States (US). There were papers from Australia ( n  = 4), South Africa ( n  = 3), Kenya ( n  = 3), Uinted Kingdom (UK) ( n  = 2), Canada ( n  = 2), China ( n  = 2), Dominican Republic ( n  = 2), Republic of Ireland ( n  = 2), Iran ( n  = 2), India (2), Honduras ( n  = 1), Netherland ( n  = 1), Vietnam ( n  = 1), Sudan ( n  = 1), New Zealand ( n  = 1), Madagascar ( n  = 1), Malaysia ( n  = 1), Ecuador ( n  = 1), Indonesia ( n  = 1), Uganda ( n  = 1), Taiwan ( n  = 1), Kyrgyzstan ( n  = 1), Saudi Arabia ( n  = 1), Haiti ( n  = 1), Honduras ( n  = 1) and Korea ( n  = 1).

Definition of needs

The review showed “need” was a multi-faceted concept with no universal definition. There was a differentiation between “health need” and “healthcare need” in the reviewed literature. Healthcare needs can benefit from health care (health education, disease prevention, diagnosis, treatment, rehabilitation and terminal care). Healthcare providers usually consider needs in terms of healthcare services that they can supply. However, health needs incorporate the wider social and environmental determinants of health, such as deprivation, housing, diet, education and employment. This broader definition allows looking beyond the confines of the medical model based on health services, to the wider influences on health [ 3 ].

In this review, relatively few empirical studies focus narrowly on healthcare needs, without attention to other determinants of health that can affect health [ 18 , 19 , 20 , 21 , 22 , 23 ]. Most of the included empirical studies looked beyond “physical health needs” to consider wider “social determinants of health” or non-medical factors that can affect a person’s overall health and health outcomes as the conditions—shaped by political, social, and economic forces—in which people are born, grow, live, work, and age [ 24 ]. Notably, the need was recognised as a “dynamic concept” whose definition will vary with time according to context and resources available to address these needs [ 16 ].

Definition of community

In general, “community” has been defined as “people with a basis of common interests and network of personal interactions grouped either based on locality or on a specific shared concerns or both” [ 25 ]. Shared common interests are particularly important as they can be assessed and, hopefully, met at a community level [ 26 ]. Importantly, community is a dynamic concept as individuals can belong to several communities at various times. In our review, community was defined by included studies, particularly those initiated by local authorities or healthcare providers (e.g., hospitals), based on geographical indicators such as county designations or based on the location of the hospital’s/facility’s/authority’s existing or potential service users. Some included empirical studies considered community based on shared interests or characteristics such as race/ethnicity, sexual orientation, or occupation. Medically underserved populations including rural areas [ 27 , 28 , 29 , 30 ], impoverished urban sectors [ 31 ], the homeless [ 32 , 33 , 34 , 35 ], persons in poverty or of low socioeconomic status, vulnerable children and families [ 18 , 28 , 36 , 37 , 38 ], the elderly [ 8 , 39 , 40 , 41 , 42 ], women and girls [ 43 , 44 , 45 , 46 , 47 ], LGBT (Lesbian, gay, bisexual, and transgender) individuals [ 48 , 49 , 50 , 51 ], displaced populations, immigrants and racial, ethnic and religious minority groups [ 12 , 19 , 36 , 42 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 ] and persons with severe and chronic health problems [ 79 ] were considered as a “community” by a number of included studies.

While defining community, a number of its characteristics were determined by included studies including: history, existing groups, physical aspects (i.e. geographic location, community size, its topography and etc.), infrastructure (i.e. health and social care facilities, public transportation, roads, bridges, electricity, mobile telephone services and etc.), demographics (i.e. age, gender, race and ethnicity, marital status, education, number of people in household, first language and etc.), economic conditions, deprivation and/or inequalities, government/politics, community leaders (formal and informal), community culture (formal and informal), existing institutions, crime and community safety, lifestyle and leisure, general health problems and epidemiology.

In our review, community health needs and assets assessment were performed by different organizations as the first step in community health promotion planning, including local health authorities (district/local), community entities [i.e. non-governmental organizations (NGOs), civil society organizations (CSOs), faith-based organizations (FBOs), community-based organizations (CBOs)] and hospitals (public/private). Included studies mostly conducted health needs assessment at the local level (e.g. cities, counties, or other municipalities). The broader understanding of health and its determinants suggests that many public and private entities have a stake in or can affect the community’s health. To engage stakeholders in the process, a number of included empirical studies ( n  = 56, 49%) sought representatives from the community that were best positioned to speak about community health based on their specific knowledge or line of work. These stakeholders were individuals from community and entities who may explicitly be concerned with health or not, which varied by the community context and culture. To have a comprehensive overview of a community needs, it was asserted that defining communities needs to be dynamic and socially constructed to take into account all voices and members, especially those not ordinarily included [ 80 ]. Community should be defined in a manner that does not exclude medically underserved, low-income, or minority populations. Integrating community voices is especially important in designing plans and programmes aimed at reducing health disparities in the community [ 58 , 81 , 82 ].

Definition of assets

Overall, there were limited definitions for “community assets” in the reviewed literature. Assets were described as resources, places, businesses, organizations, and people that can be mobilized to improve the community [ 11 , 83 ]. This includes members of the community themselves and their capabilities. Assets can therefore be described as the collective resources which individuals and communities have at their disposal, which protect against adverse health outcomes and promote health status [ 83 , 84 ].

Of 115 included empirical studies, 30 studies addressed community assets while performing community health needs assessment. A wide range of assets, from tangible resources to intangible ones, were considered that can be classified into seven broad categories as follows:

Community demographic characteristics: Literacy rates [ 13 ], youth population [ 58 , 68 ], and elderly population [ 68 ];

Natural capitals: Geographical location and natural resources [ 21 , 81 , 85 ];

Economic and financial capitals: Community business [ 12 , 81 ] community members’ income [ 21 ], and housing land ownership [ 13 ];

Community infrastructure: Level of technology/mobile phone coverage [ 13 , 21 ], transportation [ 86 ], parks and sidewalks [ 12 ], sport and recreational facilities [ 31 , 87 , 88 ], public libraries and community centres [ 88 ];

Community social and educational facilities: Non-profit and non-governmental organizations [ 59 , 87 ], media [ 89 ], educational institutions [ 12 , 31 , 81 , 90 ], faith communities [ 58 , 81 , 90 ], and community associations [ 31 ];

Community health and social facilities: Health and social facilities and providers [ 72 , 81 , 85 , 86 , 89 ], traditional medicine providers [ 72 ], and ongoing health programmes [ 13 , 87 ];

Community’s social and cultural values and resources: Tribal and community culture [ 58 , 68 , 74 , 91 ], cultural diversity [ 81 ], spirituality and religion [ 58 , 74 ], strong family bonds and values [ 59 , 74 ], strong community connections, teamwork and willingness to volunteer [ 21 , 81 , 86 , 91 ], mutual support, social support and networks [ 45 , 58 , 81 , 85 ], unity, community cohesion and collectivity [ 21 , 59 , 74 ], community capacity [ 58 ], community-led activities [ 86 , 91 ], and community values and traditions [ 68 , 74 , 86 ], resiliency [ 58 ], unifying power of communities [ 13 ], community administration units e.g. women’s committees [ 13 ], an existing group of dedicated healthcare providers [ 39 ], a group of concerned citizens [ 39 ], community safety [ 12 ], the knowledge base of the community members themselves [ 39 ] and members’ desire to be healthy [ 58 ].

Various qualitative methods such as individual interviews (one-on-one structured conversations) or focus groups (guided, structured, small group discussions) with community members, or key informants’ interviews (formal and informal conversations with leaders and stakeholder groups) or a combination of these methods were reported as the main methods to collect information on community’s assets among reviewed studies. Of these, focus group was the widely used method in community assets assessment [ 8 , 21 , 31 , 45 , 58 , 59 , 67 , 81 , 82 , 85 , 87 , 90 , 92 , 93 ].

Definition of community health needs (and assets) assessment

The terms “Community Needs Assessment (CNA)”, “Community Health Needs Assessment (CHNA)”, and “Community Health Needs and Assets Assessment (CHNAA)” were used interchangeably in the literature referring to the process of identifying health needs (and assets) of a given community. Since this review focuses on both community needs and assets, we will use the CHNAA term for the description of the process in this paper.

None of the papers reviewed provided a specific definition for CHNAA. In general, reviewed papers defined CHNAA as: A collaborative, community-engaged, systematic, ongoing, continuous, proactive, comprehensive, cyclical, regular, modifying method or process [ 28 , 33 , 69 , 92 , 94 , 95 , 96 , 97 , 98 ]; For the identification, collection, assembly, analysis, distribution, and dissemination of information on key health needs, social needs, concerns, problems, gaps, issues, factors, capabilities, strengths, assets, resources; About communities (or individuals) [ 21 , 23 , 28 , 31 , 33 , 37 , 41 , 45 , 54 , 79 , 89 , 94 , 95 , 96 , 97 , 99 , 100 , 101 , 102 ]; To achieve agreed priorities, create a shared vision, plan actions, garner resources, engage stakeholders, work collaboratively, establish relationships, implement culturally appropriate, multi-sectoral/multilevel intervention strategies, empower residents and enhance community capacity and participation in decision-making process [ 12 , 13 , 20 , 27 , 28 , 37 , 45 , 70 , 79 , 89 , 91 , 92 , 94 , 95 , 97 , 98 , 99 , 101 , 102 , 103 , 104 ]; Towards improving health and wellbeing, building and transforming health of the communities, increasing community benefits, reducing inequalities; Through which primary/secondary healthcare can respond to local and national priorities [ 20 , 23 , 28 , 40 , 51 , 59 , 69 , 97 , 103 , 105 , 106 ].

The included studies listed a number of reasons as the rationale for conducting CHNAA. Legislative requirements were most cited as the main rational for conducting CHNAA, particularly among studies conducted in the UK and US. Since the late 1980s, the concept of health needs assessment has gained increasing prominence within the National Health Service (NHS) in the UK. This has been prompted by a series of policy initiatives requiring health facilities to assess needs of their populations and to use these assessments to set priorities to improve the health of their local population [ 107 , 108 ]. In the US, several national, federal, state, and local funding sources require entities to conduct CHNAA to demonstrate a significant need for their services and programmes to be funded. The most important one is Patient Protection and Affordable Care Act (ACA-2010), requiring non-profit hospitals as tax-exempt entities to perform CHNAAs to maintain non-profit status regularly [ 92 ]. Other reasons were mentioned by included studies as the rationales for conducting CHNAA were: lack of information of health needs of a specific community, to facilitate health research and related interventions in a community, to inform the design of contextually relevant programmes and policies, to develop community health improvement plans or health promotion interventions, to develop or update strategic plans, and to receive resources and funds.

Key steps to conduct CHNAA

The number and nature of CHNAA process steps varied among reviewed studies. However, broadly CHNAAs involved six main steps as follow:

Formulation of a leadership team

Forming a leadership team, which was called by different names such as the steering committee/ the research advisory committee (RAC)/ the collaborative task force/ or the community advisory board (CAB), was known as the preliminary step of a CHNAA process. The steering committee was usually composed of local representatives from local agencies and organizations (e.g. non-profit organizations, community service agencies, media outlets, county and municipal governments, colleges and universities, faith-based organizations, and healthcare providers), community members, community stakeholders and leaders, academic partners, health and social officials, and representatives from the investigator body to help guide the development of the CHNAA project.

Leadership team responsibilities were reported as providing inputs on the research purpose, selecting and verifying study methodology and design, providing inputs and feedback on initial survey/topic content and selecting final survey/ topic guide questions, reviewing survey/topic guide length, and ensuring culturally relevant and resonant wording, comprehension and face validity, and monitoring the progress of the data collection. Feedback and recommendations from the steering committee were incorporated throughout the CHNAA process as well. Steering committees usually met on a regular basis.

Identification of needs, assets and prioritisation

To collect information on community health, needs and assets, both primary and secondary data were utilized by included studies. Secondary data included information on community socio-demographic and indicators on health status, access, utilization and satisfaction with health and social services at different levels (e.g. community, sub-national and national) to develop a picture of the overall community health. Primary data were collected through quantitative and qualitative methods and mixed-methods approaches.

Quantitative studies 

Some empirical studies used individual/household surveys as the only source to identify community needs and concerns ( n  = 28, 24.%). Surveys were a popular method of gathering opinions, preferences and perceptions of needs. Needs assessment surveys typically have written, closed-ended questions filled through the interview (face to face/telephone) or self-completion (paper or online) by community members. Generally, two main kinds of surveys were used by included studies: a) community health assessment survey, and b) community concerns survey. A number of included studies used health assessment surveys as the key data sources of the CHNAA process ( n  = 22, 19%) or along with other types of data, mainly qualitative data ( n  = 21, 18.%). Health assessment surveys typically collected information on demographics, socio-economic variables, respondents’ health status, choice of healthcare providers, and healthcare access issues among community members. Survey questionnaires were mostly developed with inputs from the literature review (similar health assessment surveys conducted at the local or national level), community members and project team discussions. Additional file  5 shows the most important data and indicators collected by included studies through conducting community health  assessment surveys.

Another form of surveys, used alone or in combination with qualitative methods ( n  = 15, 13.5%), was the community concerns survey in which people (community members and/or key informants) are asked to help identify what they see as the most important issues facing their community leading to an inventory of their health priorities [ 12 , 20 , 23 , 27 , 29 , 55 , 69 , 74 , 101 , 103 , 109 , 110 , 111 , 112 , 113 ]. A straightforward way to estimate the needs of a community was to simply ask residents their opinion on what particular services are most needed in the community. The focus of this methodology was to create an agenda based on the perceived needs and concerns of community residents. The concerns surveys were based on either focus group discussion with community members and experts or literature review by the researchers or both. Generally, while filling community concerns survey, individuals were asked to rate the importance of each issue in their community on a scale (e.g. 0 = not important, 5 = extremely important) [ 23 , 27 , 29 , 55 , 74 , 110 ]. Participants could also add and rate concerns or service needs that were not listed. Finally, each health problem identified by the community was weighted based on the frequency it was selected on the survey.

General coverage of the surveys was the population aged 18 or over currently residing in the community for a minimum period of time (at least a few months) and able to provide consent for participation. Most surveys were written, closed-ended questions filled through face to face or telephone interviews or self-completion by community members. In addition to the paper-form survey, some studies used email and social media platforms to allow residents to anonymously complete online surveys [ 29 , 51 , 57 , 96 , 103 , 110 , 114 ]. A few studies reported that residents received monetary or nonmonetary incentives for their participation upon survey completion [ 19 , 71 , 74 , 77 , 110 ]. Sampling techniques commonly used are those that promote participation in CHNAAs such as convenience sampling [ 20 , 35 , 40 , 51 , 52 , 57 , 64 , 65 , 71 , 74 , 75 , 77 , 86 , 96 , 101 , 103 , 104 , 110 , 114 , 115 ]. Only a few studies used random sampling or demonstrated the representativeness of their samples. Their response rates varied between 8 to 95.5%. Most surveys recruited local surveyors and provided them with research training to ensure consistent survey administration to attract community participation. Some studies that assessed health needs among immigrant communities or minority groups recruited bilingual surveyors or/and provided participants with two versions of the instruments, one in the native language to maximize community engagement [ 12 , 27 , 52 , 65 , 71 , 86 , 103 ]. Surveys that took a participatory approach to the design, content, terminology, and language level, were reported more understandable and culturally relevant to the community members [ 52 , 65 , 75 ].

Health needs assessment surveys (both concerns surveys and health assessment surveys) reported limitations to data collection based on the assessment timing, data availability, and sample response. As said earlier, using a convenience sampling and non-representative samples, small sample size and inter-rater reliability between surveyors were among some important methodological limitations reported by these studies, which limited the generalisability of the study findings to the entire community population [ 35 , 57 , 65 , 71 , 74 , 75 , 77 , 96 , 106 , 116 ]. Convenience sampling method and using community events as sampling sites led to sampling bias in some studies (e.g., an over-representation of some specific groups of the population such as women and low –income or high-income groups) [ 57 , 63 , 65 , 66 , 71 , 74 , 75 , 78 , 103 , 114 , 115 ].

Qualitative studies

Among included studies, about 34% ( n  = 39) used qualitative methods as the main source of data collection on community needs and assets. Some of these studies justified the use of qualitative approach by explaining how the overreliance on quantitative, population-level data resulted in CHNAAs failing to identify health needs and interests of all community members, particularly those of vulnerable population and underrepresented marginalized segments of the community. In addition, these studies concluded that integrating qualitative methods into the CHNAA process has the potential to involve community members in a more participatory fashion, perhaps improving future collaborations between communities and service providers. Such collaborations can help to design focused initiatives, making them more meaningful and culturally appropriate [ 12 , 59 , 91 , 102 ].

Key informant interviews, individual interviews with community members, focus groups with community members and community forums were among the qualitative data collection techniques used individually or in combination with each other by these studies to collect data on community needs and assets. They asserted that qualitative techniques specifically targeted to underrepresented segments of the population proved to be effective mechanisms to explore the participants’ perceptions on issues surrounding community health needs and assets. The most used technique to elicit community members’ opinions were focus group discussions and key informant interviews.

Small sample size and single-site setting were mentioned as the most cited limitations of  the qualitative CHNAAs that limit these studies generalisability. Because the studied communities were unique communities with unique assets, constraints, and health needs, the CHNAA findings cannot be generalised to other communities [ 32 , 39 , 62 , 70 , 72 , 73 , 91 , 117 , 118 ]. Another limitation mentioned by some studies was that the demographic composition of the focus group participants, specifically with regards to race, gender, socio-economic status and age group, did not fully reflect the population of studied community as a whole [ 13 , 61 , 62 , 72 , 97 , 119 ]. Some studies reported that they could not include all influencing key informants in the community to facilitate broader understandings of health needs [ 13 , 120 ].

Mixed- methods studies

A variety of data collection methods were used in a number of included studies to ensure that a comprehensive picture of community health needs and resources was obtained ( n  = 48, 42%). Some of these studies were two-phase explanatory mixed-methods studies, with the quantitative phase preceding the qualitative phase ( n  = 14, 12%). They conducted targeted focus groups or community listening sessions or interview with community members/key informants following needs assessment survey to supplement the findings from the survey and provide further information about health status, needs of daily living, barrier to health and access to community resources [ 8 , 21 , 41 , 53 , 55 , 66 , 67 , 93 , 94 , 95 , 99 , 113 , 114 , 121 ]. In addition to these studies, some studies used triangulation mixed-method design to obtain complementary qualitative and quantitative data on community health needs and issues ( n  = 13, 11%). These studies confirmed that using multiple data sources ensured researchers obtain a complete picture of the community health needs. Applying qualitative methods in the form of focus groups and semi-structured interviews enabled exploration of problems and needs within their social context and provided a wider perspective on issues raised. However, to conduct such studies CHNAA teams had to have members who have qualitative and quantitative expertise. There were some limitations specific to the mixed-method studies, including lack of rigor in integrating qualitative and quantitative findings, relying heavily on quantitative data for health need determination, and absence of the voices of the communities most in need [ 69 , 91 ].

Data analysis and interpretation

Qualitative data from focus group discussions and key informant interviews were mainly audio-recorded and transcribed verbatim by the research team and all identifying information was removed. Different analytical approaches, mostly content analysis and thematic analysis, were used to identify main themes related to assets, needs and gaps in the service system and priority populations.

Quantitative data from surveys were analysed using statistical software. Descriptive statistics were used to describe the sample in terms of socioeconomic background and present the prevalence of chronic diseases, risk factors, and health behaviours. Statistical analytical tests were also used to compare results between different groups of community members. Results also were compared by those at the state/ national level or from a similar community. Those diseases or risk factors that had a high prevalence among community members are regarded as priorities that to be addressed further.

Formulation of recommendations across various levels (individual, institution, community, policy levels)

Following analysis of the quantitative and qualitative data, the studies included in the review provided a thorough list of health needs and assets of the community. Included studies mainly used CHNAA outputs: 1) as a resource to provide baseline data of community’s health; 2) as a resource to prioritize and plan services; 3) as a resource for writing grant applications; 4) as a resource to guide a comprehensive health promotion strategy.

Not all included CHNAAs proposed interventions to address identified needs and issues. Some of the included studies ( n  = 45, 39%) just provided a snapshot of the most important issues faced by the studied community. They demonstrated several areas where CHNAAs provide more information to researchers, community organizations, and policy-makers. On the other hand, not all identified issues and needs were addressed by those studies performed CHNAA in order to implement interventions or strategies. In practice, specific populations or a number of specific health conditions or health risks, or overarching issues such as health inequality and disparities were prioritized by these studies.

In most cases, decisions on implementation were carried out by the CHNAA steering committees or the research teams. Only a number of studies used a clear and explicit set of criteria for deciding the importance of each issue [ 22 , 27 , 43 , 67 , 94 , 118 , 122 ]. A wide range of criteria were used by included studies such as: impact, urgency, community concern, achievability within the set time [ 94 ], seriousness, urgency, solvability, and financial burden of the problems [ 27 ], perception of survey participants on importance of the identified issues and feasibility of intervention, prevalence, fatality, social and cultural stigma [ 22 ], possible interventions, organizational capacity, and community assets and resources [ 13 ], importance and possibility of the effecting change [ 43 ], prevalence, impact on the duration of sickness, impact on mortality, and the availability of treatment [ 122 ], impact of the problem on the overall wellness, quality of life, and resources of their community [ 118 ], factors of health issue, size, seriousness, and effectiveness of available interventions [ 101 ], importance and feasibility [ 67 ].

Different techniques for ranking priorities were applied by included studies such as: 1) Multi-voting technique (decide on priorities by agreeing or disagreeing in group discussions and continuing process/rounds until a final list is developed), 2) Strategy lists (determine if the health needs are of “high or low importance” by placing emphasis on problems whose solutions have maximum impact, with the possibility of limited resource), 3) Nominal group technique (rate health problems from 1 to 10 through group discussion), and 4) Prioritization matrix (weigh and rank multiple criteria for prioritization with numeric values to determine health needs with high importance).

Overall, health priority types were categorized into four main categories by included studies:

Medical conditions (e.g. obesity, diabetes, heart diseases, asthma, mental health disorders, substance abuse, vision/ dental problems, HIV/AIDS and sexually transmitted diseases, injuries and health consultations).

Health behaviours (e.g. physical activity, eating habits/ nutrition, tobacco consumption, teen pregnancy and violence/gangs).

Community conditions (e.g. poverty and unemployment, environmental and infrastructural conditions, such as air quality/pollution, transportation, access to clean water and sanitation, community collaboration, and access to healthy food, exercise facilities and occupational concerns).

Health systems priorities (e.g. access to care, including primary care and higher levels of care, specialty care, mental/ behavioural health care and dental care, quality and acceptability of health services, lack of cultural competence in health systems, flexible hours and waiting time).

However, guided by a community-based participatory research (CBPR) approach, a number of studies involved community members and stakeholders in priority identification or ranking [ 12 , 21 , 22 , 23 , 27 , 29 , 31 , 36 , 41 , 43 , 49 , 53 , 55 , 56 , 58 , 59 , 60 , 62 , 63 , 68 , 70 , 74 , 86 , 87 , 88 , 90 , 92 , 99 , 100 , 103 , 104 , 110 , 114 , 117 , 118 , 119 , 121 , 122 , 123 , 124 , 125 , 126 , 127 , 128 , 129 ], in potential strategy selection [ 13 , 19 , 67 , 82 , 89 , 130 ], and in carrying out strategies [ 8 , 37 , 69 , 81 , 93 , 105 , 113 ]. They asserted that by involving the perspectives of the relevant stakeholders, a comprehensive overview of the issues and possible effective solutions was created.

Planning of programmes and interventions, implementation and evaluation

The results of CHNAA were used in various ways by included studies. In some studies, particularly researcher-led studies with limited support or involvement of the local authorities, CHNAA just led to the identification of new, locally relevant issues and priorities without any further actions ( n  = 45, 39%). The results of these CHNAAs provided more information to researchers, community organizations, and local policy-makers. Their results also may guide further research agenda in the community [ 18 , 21 , 23 , 29 , 35 , 39 , 40 , 42 , 44 , 48 , 49 , 50 , 52 , 54 , 55 , 62 , 64 , 65 , 66 , 69 , 70 , 71 , 72 , 73 , 76 , 77 , 78 , 85 , 96 , 106 , 122 , 123 , 131 , 132 , 133 , 134 , 135 ]. Some of these studies tried to present their results to the local authorities through various channels in the hope that it would modify existing programmes or implement new ones to meet the needs of the community residents. In addition to identification of relevant issues and priorities, included studies listed at least one outcome associated with the reported CHNAA activity as follows:

Development or modification of health and social policy and programmes: The knowledge provided by CHNAAs helped develop better tailored, and thereby potentially more effective interventions by a number of studies. Further, the information gathered from the CHNAA process was used as the baseline against which to measure future targets for assessment efforts and progress in areas were targeted ( n  = 36).

Formation of new partnership: In some cases, a new partnership among entities involved in CHNAA was formed to address health issues. One of the partnerships reported successful was the community–academic partnership in which communities used the research capacity of academic institutions to conduct the CHNAAs ( n  = 20). Another type of the partnership reported by some studies was the collaboration among healthcare organizations serving the same geographic area to conduct CHNAA jointly. Conducting a joint CHNAA may avoid duplication of planning efforts and obviate the creation of multiple community health needs assessments for the same population ( n  = 5).

Development of new recommendations: Several suggestions were proposed to be considered while designing health improvement interventions in the future by some of the included studies ( n  = 18).

Setting or altering strategic direction: Strategic agency direction was established or altered in some cases, which might indicate that the CHNAA was used to redirect resources better to meet the needs of the community ( n  = 4).

Raising awareness about health issues: One of the most important insights brought by CHNAA findings was the recognition of the health priorities and contributing factors by the community members, leaders and researchers, leading to an increased awareness of community issues among them ( n  = 8).

Engaging and motivating policy-makers and stakeholders: A few studies reported that CHNAAs provided health organizations with the opportunity to identify and interact with key policy-makers, community leaders, and key stakeholders about health priorities and concerns, which might foster a sense of collective ownership and trust in the results and increase the likelihood that the CHNAA will be used ( n  = 5).

Having an impact on obtaining resources and resource allocation: The CHNAAs provided the community partners with locally relevant information regarding the current status of health and perceived community needs to inform resource allocation and applications for new grants for the initiation of new programmes ( n  = 14)

Contribution to the development of CHNAA process: Some studies reported that the specific methods used in their CHNAA processes could contribute to more relevant and effective community health need assessment process ( n  = 10).

Dissemination of findings

Disseminating of the findings and knowledge gained to all partners involved was a foremost step of CHNAAs. The most cited product of the CHNAA process in the included studies was the community needs assessment report. This report includes information about the health of the community as well as the community’s capacity to improve the lives of residents. The report provides the basis for discussion and future actions. In addition to the final report, other channels to disseminate CHNAAs findings were reported as: publishing CHNAA main results in local newspapers, communicating research results with community members and stakeholders in public forums or meetings, presentation results to the steering committee and various stakeholders, posting the report on the local authorities websites, individual meetings with community leaders and stakeholders, posters, and presentation of findings in academic conferences.

Community participation

Among included studies, around 50 studies (44%) reported using participatory approaches and techniques to encourage community members' participation in CHNAA process. Unlike traditional approaches to health needs assessment, participatory approaches aimed to incorporate community inputs at all stages of the research process to enhance capacity building and overcome barriers to research raised by matters of trust, communication, cultural differences, power and representation. A variety of participatory approaches (e.g. community based participatory research (CBPR), participatory rural appraisal, participatory action research (PAR), rapid participatory appraisal (RPA), tribal participatory research, community-based collaborative action research (CBCAR), precede-proceed model, concept mapping and photovoice) were used by these studies to ensure that communities participate in CHNAA, from defining the community to identifying needs and assets and developing new interventions.

Pennel and colleagues classified the depth of the community participation in CHNAA activities into four main categories [ 136 ]. In this classification, depth of the community participation was assessed by the types of activities in which participants were involved throughout the assessment and planning process as follows:

No participation: No attempt to engage community stakeholders or members;

Consultation-only: Engagement of health-related stakeholders, broader community stakeholders, and/or community members to identify health needs through surveys, interviews, and/or focus groups; verified or validated health needs/priorities with local experts;

Moderate participation: Involvement of community stakeholders/ or community members in priority identification; involvement of community stakeholders in strategy selection;

Extensive participation: Involvement of community stakeholders/or community members to develop and carry out strategies.

The above classification was used to assess the depth of the community participation by included studies. Based on the content analysis, community participation in CHNAA process varied considerably across the included empirical studies, from minimal to in-depth participation (Table 1 ). Around 65% of the included studies were involved in consultation-only to identify health needs through one-way communication using tools such as surveys, interviews, and focus group to identify community needs and resources. Around 22% of the included studies solicited moderate participation from the community by involving community in verifying needs and final priority selection and only about 10% of the included studies reported a broad and deep community participation including community involvement in designing and implementing strategies to improve community health.

Three categories of challenges were cited by the reviewed studies while performing CHNAA projects.

Methodological challenges: These are mainly associated with quantitative and qualitative data collection methods, which were discussed earlier. Other methodological challenges cited were: difficulties in aggregating and making sense of data collected from various sources (triangulation), non-generalisability of site-specific data and limitations of the use of existing epidemiological data alone, which does not provide a comprehensive view of health needs, yet is often the most available source of information. Traditional approaches to data collection were challenging where language and literacy barriers existed [ 12 , 52 , 65 , 71 ]. Another major challenge reported by studies used community-based participatory research approaches was the challenge of involving the community in decisions related to research design and data collection methods while maintaining an appropriate level of methodological validity and reliability [ 56 , 81 , 121 ]. In addition, participation was not without challenges. Including the perspectives of stakeholders and residents can lead to differing accounts of what services are seen as essential, and each party may push their own agenda based on their personal or professional interests. Further, linguistic and cultural barriers may be a major factor among minority groups hindering participation in such endeavors [ 81 , 137 ].

Logistical challenges: The major logistical challenges reported were the need for a considerable amount of time (often inadequate), and resources required to conduct a comprehensive assessment [ 80 , 138 ]. Good quality local data on the needs and utilization of health services are usually difficult to obtain [ 9 ]. Financial costs are considerable and the depth of information obtained will ultimately depend upon the methods employed [ 139 , 140 ]. In addition, health professionals, managers and others involved in health services planning and delivery may not have the requisite skills to conduct CHNAAs. This goes beyond technical skills and places an emphasis on soft skills and flexibility including good listening skills, the ability to establish trusting relationships, empathy, working with diverse groups and reflexivity [ 140 , 141 ]. Moreover, limited health information infrastructure and systems in developing countries settings may have hindered the availability of good quality information to conduct CHNAAs [ 13 , 28 , 30 , 142 ].

Ethical challenges: Concerns were raised about the ethical issues associated with community consultation about felt needs followed by priority setting process that leaves many needs unaddressed and the bulk of expectations dashed. Labelling, stigma and stereo- typing are other problems raised by needs assessment [ 143 ]. Needs assessment results may not be utilised, leaving unmet expectations and may require extensive financial and political support to lead to changes in health service planning and delivery [ 9 ]. Comprehensive health needs assessment is likely to produce different, potentially conflicting needs, exposing hidden conflicts and tensions in communities without any mechanisms to address these issues [ 5 ]. Further, local participation may only allow those who are able to voice their needs to do so, leaving behind the silent or hidden voices [ 81 ]. Involvement of the community in the needs assessment process also impacts upon possible outcomes of the project especially since it is likely that expectations of changes to programmes and service delivery may have arisen from local participation [ 144 ].

Facilitators and enablers

CHNAA projects need to be organized in such a way that they have clear objectives, and are adequately resourced by experienced staff. In addition, factors such as clear objectives, decisive leadership, teamwork, communication, sound study design, adequate resourcing, skilled staff, sufficient time and ownership by stakeholders are among those factors that contribute to the successful implementation of CHNAAs [ 15 , 145 ]. Most studies cited community participation as a major facilitator of the CHNAA process and outcomes. Participation was shown to foster bidirectional learning and communications, where both health authorities and the community learnt about needs and priorities. Different benefits for community engagement were mentioned by reviewed literature including, improved participants’ recruitment, enhanced capacity among stakeholders, productive conflict resolution, increased quality of outputs and outcomes, increased sustainability of project goals beyond funding and timelines and development of linguistically and culturally appropriate measures. In addition, incorporating community voices has the potential to inform the development of sound measures to tackle health disparities in the basis of race, social class and ethnicity [ 12 , 27 , 30 , 91 , 103 , 110 , 126 , 146 ].

The main objective of our scoping review was to provide an overview of why and how community health needs and assets assessments (CHNAAs) have been used globally. Substantial variation was found among the studies reviewed concerning definitions, process, participants, methods, goals, and products, yet there were many common characteristics.

Some CHNAAs focused narrowly on health care in assessing needs, with scant attention to other community issues that can affect health. However, most of the included studies looked beyond health needs and considered social and environmental conditions influencing community health. We argue all CHNAAs should approach community health needs assessment holistically, focusing on both individual physical and mental wellbeing as well as casting a social determinants of health lens on the population health.

The review showed that community health needs assessment is used widely by different users and across different settings in both HICs and LMICs. However, in countries such as the US it has become institutionalized and has accordingly been developed, as service providers, particularly hospitals, are mandated to perform CHNAA to compliance with legislative mandates. However, though federal and state laws impose requirements on hospitals to conduct CHNAAs, the methods for needs assessments are generally left to the discretion of each hospital [ 147 ]. As a result, assessment methods vary widely. US-based CHNAAs either develop their own CHNAA processes or utilize a process developed at the state or national level to guide their efforts. A number of toolkits have been provided by different organizations across US to help healthcare providers to conduct CHNAA projects [ 6 , 148 , 149 ]. This highlights the need for consensus guidance across many countries and settings while maintaining the responsiveness to contextual needs, assets and priorities.

Both qualitative and quantitative approaches were employed to collect data on community health needs and assets. Overall, there has been a growing use of mixed-methods approaches to conduct CHNAA in recent years, owing to the recognition in the literature that using qualitative and quantitative approaches simultaneously can provide complementary insights determining community health needs and assets [ 69 , 91 , 104 ]. Although quantitative approaches yield concrete evidence of community needs and assets, qualitative approaches provide a context for how these issues can be addressed using available resources [ 91 , 102 ]. Using qualitative methods in conjunction with more traditional quantitative approaches is especially appropriate for studying complex public health issues and promotes the alignment of implementation plans with the local needs of community members [ 59 , 69 , 91 ]. The growing use of mixed-methods approaches has practical implications for research training and capacity building within entities performing CHNAAs. Organizations who wish to conduct CHNAAs will need to ensure that the competencies and expertise required for mixed-methods studies are available.

Although only a small number of studies provided definitions of assets, there is a growing interest in the literature in asset-based assessment, which examines and mobilizes community assets, instead of focusing on only the needs of communities [ 11 , 84 ]. Unlike need-based or deficit approaches, asset-based approaches document resources and focus on strengths to enhance and preserve rather than deficits to be remedied. Related to principles of empowerment, it postulates that solutions to community problems already exist within a community’s assets. By recognizing existing capacity, communities can become empowered to take ownership of their health and improve as a population [ 11 , 31 , 125 ]. An asset-based approach was recognized as essential for enhancing trust and community coalitions [ 83 ]. Further, it is more participatory in nature through involving community stakeholders throughout the needs assessment process [ 82 , 83 ]. In particular, it highlights community resilience, resources, and opportunities for positive growth rather than focusing solely on health problems or other concerns [ 14 , 84 , 88 ]. In developing countries, assets identified from within the community are crucial for later use in the implementation of health programmes. The shift from a traditional needs-based perspective to an asset-based perspective to health needs assessment can help to address resource constraints in these countries [ 13 , 30 , 150 ].

There was a growing interest in the use of participatory approaches and in their value in identifying and addressing community health needs over recent years among included studies. About half of the reviewed studies applied CBPR or other community-engaged approaches to perform CHNAA. There are several opportunities to fully engage patients, families, and communities in healthcare delivery redesign to ensure that they are provided in a way that address the community members’ needs and preferences. The CHNAA process is one mechanism for this engagement—and a good precursor to deeper engagement and collaboration [ 91 , 97 , 123 ]. Integrating community voices into CHNAA process may be crucially important for confronting health disparities at the community level, which stemming from socio-historical processes, including racial and ethnic discrimination and economic inequality [ 33 , 74 , 86 , 91 ]. To eliminate health disparities, it is critical first to understand social, cultural, and economic determinants of health. CHNAAs, particularly when they include the voices of community residents, can provide an opportunity to understand local processes contributing to health disparities. This knowledge can then be used to inform health and equity initiatives [ 91 , 110 , 126 ]. The development process and implementation of a CHNAA project is an important example of evidence-based public health practice. It is a way to address health and health care disparities experienced by medically underserved populations [ 86 , 92 , 126 ]. Those studies used a participatory approach reported that by having community participation, concerns and issues of the most marginalized and vulnerable populations were voiced. The inclusion of these voices allowed for a broader and deeper understanding of the concerns of those who are typically marginalized and that may be missed in traditional health needs assessment methodologies [ 33 , 56 , 58 , 74 , 86 , 110 , 137 , 146 ]. Hence, defining communities while performing CHNAA needs to be dynamic and socially constructed to take into account all voices and members especially those not ordinarily included. This deeper understanding is critical to move public health practice and research upstream to address structural and social determinants of health necessary for population-level reductions in health inequities [ 80 , 91 ].

Although there is widespread theoretical recognition of the importance of in-depth community participation in CHNAA, this has not been fully embraced in practice based on our review. Included studies reported community involvement in various stages of CHNAA with varying depth reflecting a continuum from no participation to extensive participation, in which most studies were located at the middle of the participation continuum. The literature review suggests while certain community stakeholders were engaged in the CHNAA process, most studies did not involve a broad range of stakeholders through adopting a full participation approach. One reason for this could be that for most studies conducted in the US, CHNAA was performed to comply with ACA requirements, which requires hospitals to incorporate inputs of the population served as part of the CHNAA process. Since community inputs as well as the process as a whole is not well-defined by these regulations [ 20 ], it seems that the majority of included US-based studies tried to meet legislative requirements by incorporating a minimum level of community and stakeholders’ participation in CHNAA process. In addition, the concept of community engagement in health services planning and implementation has evolved over recent years, from one-way consultative processes to bi-directional collaboration and shared leadership. Although undertaking an in-depth participatory approach through extensive participation of community stakeholders in CHNAAs may pose certain challenges for healthcare providers including requiring additional time and other resources to collaborate with community residents, we argue the benefits to this approach are important to improve health, as reported by some included studies [ 80 , 118 , 151 ].

A notable gap in the existing literature is the lack of long-term or longitudinal–assessment of CHNAA. The review showed that additional research into CHNAA implementation and outcomes is needed. Currently, there are limited data describing the impact of CHNAAs on health outcomes. However, there is ample evidence on different short-term impacts associated with CHNAA implementation, including, the development of health and social interventions, forming the new partnership, raising awareness on health issues, engaging policy-makers, and facilitating obtaining resources. In other words, it is unclear how CHNAA projects are linked directly to health outcomes. Furthermore, the mechanisms between the conduct and use of CHNAA remain largely unknown in the literature [ 152 , 153 ]. Clearly, not all CHNAA projects result in changes to policies or programmes, and conversely, many programme and policy decisions are made in the absence of CHNAA data [ 154 , 155 ]. Still, further research to understand these mechanisms and the long term impact of CHNAA is needed to support evidence of its use and value in addressing individual and population health needs.

This scoping review aimed to provide clarity and supplement the evidence on the key concepts, rationale, methods, tools and outcomes of community health needs and assets assessments (CHNAAs). Importantly, it highlights the need for holistic approaches to needs assessments to focus on physical, mental and social wellbeing, along with considering wider systems factors and structural challenges to individual and population health. Furthermore, the findings emphasize the inclusion of community assets in community health assessments, beginning foremost with community capabilities and knowledge. It is encouraging to see the use of pragmatic approaches including both qualitative and quantitative methods in CHNAA process in the literature. This will help to ensure that a robust and in-depth exploration of needs and assets is available to guide decision making. Although we recognize the challenges with providing consensus on definitions, processes and tools for CHNAA, we argue that more clarity is needed on the key considerations, steps and outcomes for this process across various settings. This study attempts to provide some theoretical insights and empirical information concerning the process, which hopefully will provide useful guidance to community organizations, policy- makers, health service providers and researchers seeking to develop and implement community health needs and assets assessment.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its supplementary information files.

Institute of Medicne. The future of the public’s health in the 21st century: National Academy Press; 2003.

Google Scholar  

Fuller J, Bentley M, Shotton D. Use of community health needs assessment for regional planning in country South Australia. Aust J Rural Health. 2001;9(1):12–7. https://doi.org/10.1046/j.1440-1584.2001.00353.x .

Article   CAS   Google Scholar  

Wright J, Williams R, Wilkinson JR. Development and importance of health needs assessment. BMJ. 1998;316(7140):1310–3. https://doi.org/10.1136/bmj.316.7140.1310 .

Wright J, Walley J. Assessing health needs in developing countries. BMJ. 1998;316(7147):1819–23. https://doi.org/10.1136/bmj.316.7147.1819 .

Jordan J, Dowswell T, Harrison S, Lilford RJ, Mort M. Health needs assessment: Whose priorities? Listening to users and the public. BMJ. 1998;316(7145):1668–70. https://doi.org/10.1136/bmj.316.7145.1668 .

Myers S, Stoto MA: Criteria for assessing the usefulness of community health assessments; a literature review. 2006. https://www.rand.org/pubs/technical_reports/TR314.html . Accessed 18 Nov 2020.

Centers for Disease Control and Prevention (CDC). Community needs assessment. Atlanta: GA.

Finifter DH, Jensen CJ, Wilson CE, Koenig BL. A comprehensive, multitiered, targeted community needs assessment model: methodology, dissemination, and implementation. Fam Community Health. 2005:293–306. https://doi.org/10.1097/00003727-200510000-00003 .

Stevens A, Gillam S. Needs assessment: from theory to practice. BMJ. 1998;316(7142):1448–52. https://doi.org/10.1136/bmj.316.7142.1448 .

Friedman DJ, Parrish RG. Is community health assessment worthwhile? J Public Health Manag Pract. 2009;15(1):3–9. https://doi.org/10.1097/01.PHH.0000342943.41080.ef .

Article   Google Scholar  

Altschuld JW, Hung HL, Lee YF. Needs assessment and asset/capacity building: A promising development in practice. New Dir Eval. 2014;2014(144):89–103. https://doi.org/10.1002/ev.20105 .

Jackson KM, Pukys S, Castro A, Hermosura L, Mendez J, Vohra-Gupta S, et al. Using the transformative paradigm to conduct a mixed methods needs assessment of a marginalized community: Methodological lessons and implications. Eval Program Plann. 2018;66:111–9. https://doi.org/10.1016/j.evalprogplan.2017.09.010 .

Mathias KR, Mathias JM, Hill PC. An asset-focused health needs assessment in a rural community in North India. Asia Pac J Public Health. 2015;27(2):NP2623–34. https://doi.org/10.1177/1010539511421193 .

Kirk CM, Johnson-Hakim S, Anglin A, Connelly C. Putting the community back into community health needs assessments: Maximizing partnerships via community-based participatory research. Prog Community Health Partnersh. 2017;11(2):167–73. https://doi.org/10.1353/cpr.2017.0021 .

Coster G, Mays N, Scott C, Cumming J. The impact of health needs assessment and prioritisation on District Health Board planning in New Zealand. Int J Health Plann Manag. 2009;24(4):276–89. https://doi.org/10.1002/hpm.1011 .

Parry-Jones B, Soulsby J. Needs-led assessment: the challenges and the reality. Health Soc Care Community. 2001;9(6):414–28. https://doi.org/10.1046/j.0966-0410.2001.00316.x .

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. JBI Evidence Implementation. 2015;13(3):141–6. https://doi.org/10.1097/XEB.0000000000000050 .

Anselma M, Chinapaw MJM, Altenburg TM. Determinants of child health behaviors in a disadvantaged area from a community perspective: A participatory needs assessment. Int J Environ Res Public Health. 2018;15(4):644. https://doi.org/10.3390/ijerph15040644 .

Berkley-Patton J, Thompson CB, Bradley-Ewing A, Marcie B, Booker A, Catley D, et al. Identifying health conditions, priorities, and relevant multilevel health promotion intervention strategies in African American churches: A faith community health needs assessment. Eval Program Plann. 2018;67:19. https://doi.org/10.1016/j.evalprogplan.2017.10.012 .

Bias TK, Abildso CG, Vasile E, Coffman J. The impact of community input in community health needs assessments. J Public Health Manag Pract. 2017;23:S29–33. https://doi.org/10.1097/PHH.0000000000000586 .

Cho S, Lee H, Yoon S, Kim Y, Levin P, Kim E. Community health needs assessment: a nurses’ global health project in Vietnam. Int Nurs Rev. 2018;65(4):505–14. https://doi.org/10.1111/inr.12443 .

Kapiriri L, Frithjof NorheimMD PDO. Whose priorities count? Comparison of community-identified health problems and Burden-of-Disease-assessed health priorities in a district in Uganda. Health Expect. 2002;5(1):55–62. https://doi.org/10.1046/j.1369-6513.2002.00161.x .

Li Y, Cao J, Lin H, Li D, Wang Y, He J. Community health needs assessment with precede-proceed model: a mixed methods study. BMC Health Serv Res. 2009;9(1):1–14. https://doi.org/10.1186/1472-6963-9-181 .

World Health Organization. Social Determinants of Health. https://www.who.int/health-topics/social-determinants-of-health#tab=tab_1 . Acessed 10 Oct 2022.

Smithies J, Adams L, Webster G, Beattie A. Community participation in health promotion: Health Education Authority London; 1990.

Jack K, Holt M. Community profiling as part of a health needs assessment. Nurs Stand. 2008;22(18):51–8. https://doi.org/10.7748/ns2008.01.22.18.51.c6311 .

Ahari SS, Habibzadeh S, Yousefi M, Amani F, Abdi R. Community based needs assessment in an urban area; A participatory action research project. BMC Public Health. 2012;12(1):1–8. https://doi.org/10.1186/1471-2458-12-161 .

Shin H, Lee SJ, Lee Y-n, Shon S. Community health needs assessment for a child health promotion program in Kyrgyzstan. Eval Program Plann. 2019;74:1–9. https://doi.org/10.1016/j.evalprogplan.2019.02.005 .

Birdwell SW, Calesaric H. Identifying Health Care Needs of Rural Ohio Citizens: An Evaluation of a Two-stage Methodology. J Rural Health. 1996;12(2):130–6. https://doi.org/10.1111/j.1748-0361.1996.tb00784.x .

Pepall E, Earnest J, James R. Understanding community perceptions of health and social needs in a rural Balinese village: results of a rapid participatory appraisal. Health Promot Int. 2007;22(1):44–52. https://doi.org/10.1093/heapro/dal042 .

Moore de Peralta A, Davis L, Brown K, Fuentes M, Falconer NS, Charles J, et al. Using Community-Engaged Research to Explore Social Determinants of Health in a Low-Resource Community in the Dominican Republic: A Community Health Assessment. Hisp Health Care Int. 2020;18(3):127–37. https://doi.org/10.1177/1540415319874812 .

Ake T, Diehr S, Ruffalo L, Farias E, Fitzgerald A, Good SD, et al. Needs assessment for creating a patient-centered, community-engaged health program for homeless pregnant women. J Patient Cent Res Rev. 2018;5(1):36. https://doi.org/10.17294/2330-0698.1591 .

Kiser T, Hulton L. Addressing health care needs in the homeless population: a new approach using participatory action research. SAGE Open. 2018;8(3):2158244018789750. https://doi.org/10.1177/2158244018789750 .

Schaffer MA, Mather S, Gustafson V. Service learning: A strategy for conducting a health needs assessment of the homeless. Health Care Poor Underserved. 2000;11(4):385–99. https://doi.org/10.1353/hpu.2010.0746 .

Kehn M, Ho P, Kroll T. Identifying the health service needs of homeless adults with physical disabilities. Public Health. 2013;127(8):785–7. https://doi.org/10.1016/j.puhe.2013.01.011 .

Sangalang CC, Ngouy S, Lau AS. Using community-based participatory research to identify health issues for Cambodian American youth. Fam Community Health. 2015;38(1):55. https://doi.org/10.1097/FCH.0000000000000056 .

Jeffery V, Ervin K. Responding to rural health needs through community participation: addressing the concerns of children and young adults. Aust Prim Health. 2011;17(2):125–30. https://doi.org/10.1071/PY10050 .

Harrison L, Scarinci I. Child health needs of rural Alabama Latino families. J Community Health Nurs. 2007;24(1):31–47. https://doi.org/10.1080/07370010709336584 .

Averill J. Keys to the puzzle: Recognizing strengths in a rural community. Public Health Nurs. 2003;20(6):449–55. https://doi.org/10.1046/j.1525-1446.2003.20605.x .

Din NC, Ghazali SE, Ibrahim N, Ahmad M, Said Z, Ghazali AR, et al. Health needs assessment of older people in an agricultural plantation. Int J Gerontol. 2014;8(3):120–6. https://doi.org/10.1016/j.ijge.2013.12.003 .

Nolin J, Wilburn ST, Wilburn KT, Weaver D. Health and social service needs of older adults: Implementing a community-based needs assessment. Eval Program Plann. 2006;29(3):217–26. https://doi.org/10.1016/j.evalprogplan.2006.06.003 .

Lee LL, Lin SH, Philp I. Health needs of older Aboriginal people in Taiwan: a community-based assessment using a multidimensional instrument. J Clin Nurs. 2015;24(17–18):2514–21. https://doi.org/10.1111/jocn.12842 .

Núñez AE, Robertson-James C, Reels S, Weingartner RM, Bungy BL. Conducting a needs assessment for women and girls using a gender analysis framework: The Philadelphia Ujima coalition for a healthier community experience. Womens Health Issues. 2012;22(6):e527–34. https://doi.org/10.1016/j.whi.2012.08.002 .

Urrutia RP, Merisier D, Small M, Urrutia E, Tinfo N, Walmer DK. Unmet health needs identified by Haitian women as priorities for attention: a qualitative study. Reprod Health Matters. 2012;20(39):93–103. https://doi.org/10.1016/s0968-8080(12)39602-x .

Zimmermann K, Khare MM, Wright C, Hasler A, Kerch S, Moehring P, et al. Application of a gender-based approach to conducting a community health assessment for rural women in Southern Illinois. Eval Program Plann. 2015;51:27–34. https://doi.org/10.1016/j.evalprogplan.2014.12.004 .

Dell EM, Erikson SL, Andrianirina E, Smith G. Women’s knowledge in Madagascar: A health needs assessment study. Global public health. 2012;7(1):29–41. https://doi.org/10.1080/17441692.2011.557083 .

Chhabra R, Teitelman N, Silver EJ, Raufman J, Bauman LJ. Vulnerability Multiplied: Health Needs Assessment of 13–18-Year-Old Female Orphan and Vulnerable Children in Kenya. World Med Health Policy. 2018;10(2):129–45. https://doi.org/10.1002/wmh3.267 .

Stepleman LM, Yohannan J, Scott SM, Titus LL, Walker J, Lopez EJ, et al. Health needs and experiences of a LGBT population in Georgia and South Carolina. Homosex. 2019;66(7):989–1013. https://doi.org/10.1080/00918369.2018.1490573 .

Wilson BD, Neubauer LC, Park A, Abuor P, Harper GW. The sexual health needs of sexual minority women in Western Kenya: An exploratory community assessment and public policy analysis. Glob Public Health. 2019;14(10):1495–508. https://doi.org/10.1080/17441692.2019.1611895 .

Zajac C, Godshall KC. Empowerment through accessibility: Community needs assessment data for LGBTQ communities. Soc Work Public Health. 2020;35(6):483–93. https://doi.org/10.1080/19371918.2020.1798322 .

Coleman JD, Irwin JA, Wilson RC, Miller HC. The South Carolina LGBT needs assessment: A descriptive overview. Homosex. 2014;61(8):1152–71. https://doi.org/10.1080/00918369.2014.872515 .

Bhimla A, Yap L, Lee M, Seals B, Aczon H, Ma GX. Addressing the health needs of high-risk Filipino Americans in the greater Philadelphia region. J Community Health. 2017;42(2):269–77. https://doi.org/10.1007/s10900-016-0252-0 .

Bopp M, Fallon EA, Bolton DJ, Kaczynski AT, Lukwago S, Brooks A. Conducting a Hispanic Health Needs Assessment in rural Kansas: Building the foundation for community action. Eval Program Plann. 2012;35(4):453–60. https://doi.org/10.1016/j.evalprogplan.2012.02.002 .

Burns JC, Teadt S, Bradley WW Sr, Shade GH Jr. Enhancing Adolescent and Young Adult Health Services! A Review of the Community Needs Assessment Process in an Urban Federally Qualified Health Center. Health Equity. 2020;4(1):218–24. https://doi.org/10.1089/heq.2019.0108 .

Corona R, Gonzalez T, Cohen R, Edwards C, Edmonds T. Richmond Latino needs assessment: A community-university partnership to identify health concerns and service needs for Latino youth. J Community Health. 2009;34(3):195–201. https://doi.org/10.1007/s10900-008-9140-6 .

Doyle E, Rager R, Bates D, Cooper C. Using community-based participatory research to assess health needs among migrant and seasonal farmworkers. Am J Health Educ. 2006;37(5):279–88. https://doi.org/10.1080/19325037.2006.10598916 .

Ghimire S, Cheong P, Sagadraca L, Chien L-C, Sy FS. A health needs assessment of the Filipino American community in the greater Las Vegas area. Health Equity. 2018;2(1):334–48. https://doi.org/10.1089/heq.2018.0042 .

Hebert-Beirne J, Felner JK, Castañeda Y, Cohen S. Enhancing themes and strengths assessment: leveraging academic-led qualitative inquiry in community health assessment to uncover roots of community health inequities. J Public Health Manag Pract. 2017;23(4):370–9. https://doi.org/10.1097/phh.0000000000000478 .

Hebert-Beirne J, Hernandez SG, Felner J, Schwiesow J, Mayer A, Rak K, et al. Using community-driven, participatory qualitative inquiry to discern nuanced community health needs and assets of Chicago’s La Villita, a Mexican immigrant neighborhood. J Community Health. 2018;43(4):775–86. https://doi.org/10.1007/s10900-018-0484-2 .

Hernandez SG, Genkova A, Castañeda Y, Alexander S, Hebert-Beirne J. Oral histories as critical qualitative inquiry in community health assessment. Health Educ Behav. 2017;44(5):705–15. https://doi.org/10.1177/1090198117728546 .

Lee S, Ma GX, Juon H-S, Martinez G, Hsu CE, Bawa J. Assessing the needs and guiding the future: findings from the health needs assessment in 13 Asian American communities of Maryland in the United States. J Immigr Minor Health. 2011;13(2):395–401. https://doi.org/10.1007/s10903-009-9310-3 .

Martinez IL, Carter-Pokras O. Assessing health concerns and barriers in a heterogeneous Latino community. J Health Care Poor Underserved. 2006;17(4):899–909. https://doi.org/10.1353/hpu.2006.0129 .

Mannix TR, Austin SD, Baayd JL, Simonsen SE. A community needs assessment of urban Utah American Indians and Alaska natives. J Community Health. 2018;43(6):1217–27. https://doi.org/10.1007/s10900-018-0542-9 .

Montano JJ, Acosta-Deprez V, Sinay T. Assessing the health care needs of Filipino Americans in greater Long Beach. Public Admin Manag. 2009;14(1):156.

Patel VV, Rajpathak S, Karasz A. Bangladeshi immigrants in New York City: a community based health needs assessment of a hard to reach population. J Immigr Minor Health. 2012;14(5):767–73. https://doi.org/10.1007/s10903-011-9555-5 .

Su D, Garg A, Wiens J, Meyer E, Cai G. Assessing health needs in African American churches: a mixed-methods study. J Relig Health. 2021;60(2):1179–97. https://doi.org/10.1007/s10943-019-00924-5 .

Suiter SV. Community health needs assessment and action planning in seven Dominican bateyes. Eval Program Plann. 2017;60:103–11. https://doi.org/10.1016/j.evalprogplan.2016.10.011 .

Thomas LR, Donovan DM, Sigo RL. Identifying community needs and resources in a Native community: A research partnership in the Pacific Northwest. Int J Ment Health Addict. 2010;8(2):362–73. https://doi.org/10.1007/s11469-009-9233-1 .

Weathers B, Barg FK, Bowman M, Briggs V, Delmoor E, Kumanyika S, et al. Using a mixed-methods approach to identify health concerns in an African American community. Am J Public Health. 2011;101(11):2087–92. https://doi.org/10.2105/AJPH.2010.191775 .

Dong X, Chang E, Wong E, Wong B, Skarupski KA, Simon MA. Assessing the health needs of Chinese older adults: Findings from a community-based participatory research study in Chicago's Chinatown. J Aging Res. 2010. https://doi.org/10.4061/2010/124246 .

Sarsour L, Tong VS, Jaber O, Talbi M, Julliard K. Health assessment of the Arab American community in southwest Brooklyn. J Community Health. 2010;35(6):653–9. https://doi.org/10.1007/s10900-010-9260-7 .

Thein K, Zaw KT, Teng R-E, Liang C, Julliard K. Health Needs in Brooklyn's Chinatown: A Pilot Assessment Using Rapid Participatory Appraisal. J Health Care Poor Underserved. 2009;20(2):378–94. https://doi.org/10.1353/hpu.0.0140 .

Careyva BA, Hamadani R, Friel T, Coyne CA. A social needs assessment tool for an urban Latino population. J Community Health. 2018;43(1):137–45. https://doi.org/10.1007/s10900-017-0396-6 .

Johnson CV, Bartgis J, Worley JA, Hellman CM, Burkhart R. Urban Indian voices: a community-based participatory research health and needs assessment. Am Indian Alsk Native Mentl Health Res. 2010;17(1):49–70. https://doi.org/10.5820/aian.1701.2010.49 .

Puertas B, Schlesser M. Assessing community health among indigenous populations in Ecuador with a participatory approach: implications for health reform. J Community Health. 2001;26(2):133–47. https://doi.org/10.1023/a:1005281314274 .

Miller K, Yost B, Abbott C, Thompson Buckland S, Dlugi E, Adams Z, et al. Health needs assessment of five Pennsylvania Plain populations. Int J Environ Res Public Health. 2019;16(13):2378. https://doi.org/10.3390/ijerph16132378 .

Tendulkar SA, Hamilton RC, Chu C, Arsenault L, Duffy K, Huynh V, et al. Investigating the myth of the “model minority”: a participatory community health assessment of Chinese and Vietnamese adults. J Immigr Minor Health. 2012;14(5):850–7. https://doi.org/10.1007/s10903-011-9517-y .

Peak T, Gast J, Ahlstrom D. A needs assessment of Latino men’s health concerns. Am J Mens Health. 2010;4(1):22–32. https://doi.org/10.1177/1557988308327051 .

Brock KE, Allen KE, Barton C, Shapiro R, Weintraub B, Wasilewski-Masker K, et al. A Methodologic Approach to Conducting a Statewide Community Needs Assessment of Pediatric Palliative Care and Hospice Resources. J Pain Symptom Manag. 2020;60(3):531–8. e538. https://doi.org/10.1016/j.jpainsymman.2020.04.004 .

Franz BA, Skinner D, Murphy JW. Defining “Community” in community health evaluation: perspectives from a sample of nonprofit Appalachian hospitals. Am J Eval. 2018;39(2):237–56. https://doi.org/10.1177/1098214017722857 .

Clark MJ, Cary S, Diemert G, Ceballos R, Sifuentes M, Atteberry I, et al. Involving communities in community assessment. Public Health Nurs. 2003;20(6):456–63. https://doi.org/10.1046/j.1525-1446.2003.20606.x .

Van Gelderen SA, Krumwiede KA, Krumwiede NK, Fenske C. Trialing the community-based collaborative action research framework: supporting rural health through a community health needs assessment. Health Promot Pract. 2018;19(5):673–83. https://doi.org/10.1177/1524839917754043 .

Goldman KD, Schmalz KJ. “Accentuate the positive!” using an asset-mapping tool as part of a community-health needs assessment. Health Promot Pract. 2005;6(2):125–8. https://doi.org/10.1177/1524839904273344 .

Sharpe PA, Greaney ML, Lee PR, Royce SW. Assets-oriented community assessment. Public Health Rep. 2000;115(2–3):205. https://doi.org/10.1093/phr/115.2.205 .

Mathias K, Rawat M, Thompson A, Gaitonde R, Jain S. Exploring community mental health systems–A participatory health needs and assets assessment in the Yamuna valley, North India. Int J Health Policy Manag. 2020. https://doi.org/10.34172/ijhpm.2020.222 .

deValpine MG, Trull LH. Health Equity in Community Assessments: A Participatory Approach in Rural Virginia. SAGE Open. 2019;9(1):2158244019838925. https://doi.org/10.1177/2158244019838925 .

Lewis H, Rudolph M, White L. Rapid appraisal of the health promotion needs of the Hillbrow Community, South Africa. Int J Healthc Technol Manag. 2003;5(1–2):20–33. https://doi.org/10.1504/IJHTM.2003.003338 .

Talmage C, Mercado M, Yoder G, Hamm K, Wolfersteig W. Critiquing Indicators of Community Strengths in Community Health Needs Assessments. Int J Community Well-Being. 2021:1–22. https://doi.org/10.1007/s42413-020-00106-2 .

Idali Torres M. Assessing health in an urban neighborhood: community process, data results and implications for practice. J Community Health. 1998;23(3):211–26. https://doi.org/10.1023/a:1018720515644 .

Garrettson M, Walline V, Heisler J, Townsend J. New medical school engages rural communities to conduct regional health assessment. Fam Med. 2010;42(10):693.

Cain CL, Orionzi D, O’Brien M, Trahan L. The power of community voices for enhancing community health needs assessments. Health Promot Pract. 2017;18(3):437–43. https://doi.org/10.1177/1524839916634404 .

Grant CG, Ramos R, Davis JL, Green BL. Community health needs assessment: a pathway to the future and a vision for leaders. Health Care Manag (Frederick). 2015;34(2):147–56. https://doi.org/10.1097/hcm.0000000000000057 .

Robinson A, Cherry TS, Elliott M, Davis M, Bagwell G. Leveraging university-community partnerships in rural Georgia: A community health needs assessment template for hospitals. J Ga Public Health Assoc. 2016:365–72. https://doi.org/10.21633/JGPHA.5.409 .

Alfano-Sobsey E, Ledford SL, Decosimo K, Horney JA. Community health needs assessment in Wake County, North Carolina: partnership of public health, hospitals, academia, and other stakeholders. N C Med J. 2014;75(6):376–83. https://doi.org/10.18043/ncm.75.6.376 .

Waling A, Fildes D. ‘Don't fix what ain’t broke’: evaluating the effectiveness of a Men's Shed in inner-regional A ustralia. Health Soc Care Community. 2017;25(2):758–68. https://doi.org/10.1111/hsc.12365 .

Witten NA, Orr MP, Maskarinec GG. Medical School Hotline: An Online, Quantitative Community Health Needs Assessment of Hale ‘iwa and Waialua, O ‘ahu, Hawai ‘i. H Hawaii J Health Soc Welf. 2020;79(10):306.

Woodyard CD, Przybyla S, Hallam JS. A community health needs assessment using principles of community-based participatory research in a Mississippi Delta community: a novel methodological approach. Community Dev. 2015;46(2):84–99. https://doi.org/10.1080/15575330.2015.1006240 .

O’Kelly C, Cullen W, O’Kelly S, O’Kelly F, Bury G. A primary care-based health needs assessment in inner city Dublin. Ir J Med Sci. 2010;179(3):399–403. https://doi.org/10.1007/s11845-010-0483-5 .

Knox S, Chapman P, Your health—your say. A case study of community involvement in a community health needs assessment project in a regional city of Australia. Cities. 1995;12(2):111–4. https://doi.org/10.1016/0264-2751(95)00011-a .

Murray SA, Tapson J, Turnbull L, McCallum J, Little A. Listening to local voices: adapting rapid appraisal to assess health and social needs in general practice. BMJ. 1994;308(6930):698–700. https://doi.org/10.1136/bmj.308.6930.698 .

Stanley SA, Stein DS. Health Watch 2000: community health assessment in south central Ohio. J Community Health Nurs. 1998;15(4):225–36. https://doi.org/10.1207/s15327655jchn1504_4 .

Ying L, Daikun L, Jia H, Kai S, HongHong L, Hu Z. Assessment of community health needs of Chongqing residents. a qualitative study. Mil Med Res. 2008;23(5):289–99. https://doi.org/10.1016/S1000-1948(08)60056-8 .

Kwan K, Do-Reynoso V, Zarate-Gonzalez G, Goldman-Mellor S. Development and implementation of a community health survey for public health accreditation: Case study from a rural county in California. Eval Program Plann. 2018;67:47–52. https://doi.org/10.1016/j.evalprogplan.2017.11.004 .

Green S, Parkinson L, Bonevski B, Considine R. Community health needs assessment for health service planning: realising consumer participation in the health service setting. Health Promot J Austr. 2004;15(2):142–9. https://doi.org/10.1071/HE04142 .

Horne M, Costello J. A public health approach to health needs assessment at the interface of primary care and community development: findings from an action research study. Prim Health Care Res Dev. 2003;4(4):340–52. https://doi.org/10.1191/1463423603pc173oa .

Lundeen SP. Health needs of a suburban community: a nursing assessment approach. J Community Health Nurs. 1992;9(4):235–44. https://doi.org/10.1207/s15327655jchn0904_5 .

Jordan J, Wright J. Making sense of health needs assessment. Br Gen Pract. 1997;47(424):695.

CAS   Google Scholar  

Hensher M, Fulop N. The influence of health needs assessment on health care decision-making in London health authorities. J Health Serv Policy. 1999;4(2):90–5. https://doi.org/10.1177/135581969900400206 .

Asadi-Lari M, Farshad A, Assaei S, Mahdavi MV, Akbari M, Ameri A, et al. Applying a basic development needs approach for sustainable and integrated community development in less-developed areas: report of ongoing Iranian experience. Public Health. 2005;119(6):474–82. https://doi.org/10.1016/j.puhe.2004.08.014 .

Akintobi TH, Lockamy E, Goodin L, Hernandez ND, Slocumb T, Blumenthal D, et al. Processes and outcomes of a community-based participatory research-driven health needs assessment: a tool for moving health disparity reporting to evidence-based action. Prog Community Health Partnersh. 2018;12(1 Suppl):139. https://doi.org/10.1353/cpr.2018.0029 .

Yoo IY, Cho WJ, Chae SM, Kim MJ. Community health service needs assessment in Korea using OMAHA Classification System. Int J Nurs Stud. 2004;41(6):697–702. https://doi.org/10.1016/j.ijnurstu.2004.02.007 .

Louw J, Carolissen R. Community health needs, community participation, and evaluation research. Eval Program Plann. 1995;18(4):365–9.

Plescia M, Koontz S, Laurent S. Community assessment in a vertically integrated health care system. Am J Public Health. 2001;91(5):811. https://doi.org/10.2105/AJPH.91.5.811 .

Sampson G, Gearin KJM, Boe M. A rural local health department–hospital collaborative for a countywide community health assessment. J Public Health Manag Pract. 2015;21(1):23–30. https://doi.org/10.1097/phh.0000000000000088 .

Vyas AN, Chaudhary N, Ramiah K, Landry M. Addressing a Growing Community’s Health Needs: Project SAHNA (South Asian Health Needs Assessment). J Immigr Minor Health. 2013;15(3):577–83. https://doi.org/10.1007/s10903-012-9655-x .

Van Rie A, West N, Schwartz S, Mutunga L, Hanrahan C, Ncayiyana J, et al. The unmet needs and health priorities of the urban poor: Generating the evidence base for urban community health worker programmes in South Afric. S Afr Med J. 2018;108(9). https://doi.org/10.7196/SAMJ.2018.v108i9.13054 .

Dennis MK, Momper SL. An urban American Indian health clinic's response to a community needs assessment. Am Indian Alsk Native Ment Health Res. 2016;23(5). https://doi.org/10.5820/aian.2305.2016.15 .

Schutte DL, Goris ED, Rivard JL, Schutte BC, Committee CCRA. Community Health Needs Assessment in a Rural Setting. Foundation for a Community–Academic Partnership. J High Educ Outreach Engagem. 2016;20(2):85–108.

Velonis AJ, Molnar A, Lee-Foon N, Rahim A, Boushel M, O’Campo P. “One program that could improve health in this neighbourhood is _?” using concept mapping to engage communities as part of a health and human services needs assessment. BMC Health Serv Res. 2018;18(1):1–12. https://doi.org/10.1186/s12913-018-2936-x .

Torres Stone RA, Cardemil EV, Keefe K, Bik P, Dyer Z, Clark KE. A community mental health needs assessment of a racially and ethnically diverse population in New England: Narratives from community stakeholders. Community Ment Health J. 2020;56(5):947–58. https://doi.org/10.1007/s10597-020-00562-2 .

Agrusti T, Bohn J, Dunn E, Bell C, Ziegler A. The story so far: a mixed-methods evaluation of county-level behavioral health needs, policies, and programs. Soc Work Ment Health. 2020;18(3):257–73. https://doi.org/10.1080/15332985.2020.1720889 .

Palmer CA. Rapid appraisal of needs in reproductive health care in southern Sudan: qualitative study. BMJ. 1999;319(7212):743–8. https://doi.org/10.1136/bmj.319.7212.743 .

Lazenbatt A, Lynch U, O'Neill E. Revealing the hidden ‘troubles’ in Northern Ireland: the role of participatory rapid appraisal. Health Educ Res. 2001;16(5):567–78. https://doi.org/10.1093/her/16.5.567 .

Kingery FP, Naanyu V, Allen W, Patel P. Photovoice in Kenya: Using a community-based participatory research method to identify health needs. Qual Health Res. 2016;26(1):92–104. https://doi.org/10.1177/1049732315617738 .

Sabin AV, Levin PF. The Journey to Meet Emerging Community Benefit Requirements in a Rural Hospital: A Case Study. Online J Issues Nurs. 2016;21(1). https://doi.org/10.3912/OJIN.Vol21No01PPT01 .

Lightfoot AF, De Marco MM, Dendas RC, Jackson MR, Meehan EF. Engaging underserved populations in Affordable Care Act-required needs assessments. J Health Care Poor Underserved. 2014;25(1):11–8. https://doi.org/10.1353/hpu.2014.0060 .

Amendola MG, Nazario N, Sanchez V. Using CBPR to assess client needs at a social service agency. Public Health Nurs. 2016;33(2):167–75. https://doi.org/10.1111/phn.12204 .

Harmon BE, Pichon LC, Powell TW, Rugless F, McCann L, Minor LL, et al. Developing a Congregational Health Needs Assessment: Lessons Learned from Using a Participatory Research Approach. Prog Community Health Partnersh. 2021;15(1):47–58. https://doi.org/10.1353/cpr.2021.0004 .

Mulvihill BA, Pass MA, Miller TM, Mulvihill FX, Klerman LV. Collaborative needs assessment and systems development in Alabama: Process and products. Am J Prev Med. 1996;12(4):14–9.

Santilli A, Carroll-Scott A, Ickovics JR. Applying community organizing principles to assess health needs in New Haven, Connecticut. Am J Public Health. 2016;106(5):841–7. https://doi.org/10.2105/ajph.2016.303050 .

Milaat W, Bani I, Aqeely H, Makeen A, Salih A, Alnagaar M. Jazan Health needs assessment: a key informant approach. J Public Health Med. 2007;6:105–10.

Kulig JC, Wilde I. Collaboration between communities and universities: Completion of a community needs assessment. Public Health Nurs. 1996;13(2):112–9. https://doi.org/10.1111/j.1525-1446.1996.tb00228.x .

Running A, Martin K, Tolle LW. An innovative model for conducting a participatory community health assessment. J Community Health Nurs. 2007;24(4):203–13. https://doi.org/10.1080/07370010701645869 .

Smith SM, Long J, Deady J, O'Keeffe F, Handy D, O'Dowd T. Adapting developing country epidemiological assessment techniques to improve the quality of health needs assessments in developed countries. BMC Health Serv Res. 2005;5(1):1–7. https://doi.org/10.1186/1472-6963-5-32 .

Wood EA, Chapman KS, Beau de Rochars VM, Mckune SL. Community-based health needs assessment in Léogâne and Gressier, Haiti: six years post-earthquake. J Int Humanit Action. 2017;2(1):1–8. https://doi.org/10.1186/s41018-017-0026-z .

Pennel CL, McLeroy KR, Burdine JN, Matarrita-Cascante D, Wang J. A mixed-methods approach to understanding community participation in community health needs assessments. J Public Health Manag Pract. 2017;23(2):112–21. https://doi.org/10.1097/PHH.0000000000000362 .

Lillie-Blanton M, Hoffman SC. Conducting an assessment of health needs and resources in a racial/ethnic minority community. Health Serv Res. 1995;30(1 Pt 2):225.

Irani P, Bohn C, Halasan C, Landen M, McCusker D. Community health assessment: driving the need for current, easily accessible population health data. J Public Health Manag Pract. 2006;12(2):113–8. https://doi.org/10.1097/00124784-200603000-00002 .

Alberti P. Community health needs assessments: filling data gaps for population health research and management. EGEMS. 2014;2(4). https://doi.org/10.13063/2327-9214.1174 .

Skinner D, Franz B, Kelleher K. What challenges do nonprofit hospitals face in taking on community health needs assessments? A qualitative study from Appalachian Ohio. J Rural Health. 2018;34(2):182–92. https://doi.org/10.1111/jrh.12246 .

Stoto MA, Davis MV, Atkins A. Making better use of population health data for community health needs assessments. EGEMs. 2019;7(1). https://doi.org/10.5334/egems.305 .

Smith LU, Burdine JN. Community health assessment opportunities and challenges in the 21st century: Implications for professional development. J Public Health Manag Pract. 2017;23:S63–4. https://doi.org/10.1097/PHH.0000000000000601 .

Hawe P. Needs assessment must become more change-focused. Aust N Z J Public Health. 1996;20(5):473–8. https://doi.org/10.1111/j.1467-842x.1996.tb01624.x .

Jordan J, Wright J, Wilkinson J, Williams R. Assessing local health needs in primary care: understanding and experience in three English districts. Qual Health Care. 1998;7(2):83–9. https://doi.org/10.1136/qshc.7.2.83 .

Curtis DC. Evaluation of community health assessment in Kansas. J Public Health Manag Pract. 2002:20–5. https://doi.org/10.1097/00124784-200207000-00005 .

Kim S, Flaskerud JH, Koniak-Griffin D, Dixon EL. Using community-partnered participatory research to address health disparities in a Latino community. J Prof Nurs. 2005;21(4):199–209. https://doi.org/10.1016/j.profnurs.2005.05.005 .

Crossley M. Tax-exempt hospitals, community health needs and addressing disparities. Howard Law J. 2011;55:687.

Schifferdecker KE, Bazos DA, Sutherland KA, LaFave LRA, Fedrizzi R, Hoebeke J. A review of tools to assist hospitals in meeting community health assessment and implementation strategy requirements. J Healthc Manag. 2016;61(1):44.

Becker KL. Conducting community health needs assessments in rural communities: Lessons learned. Health Promot Pract. 2015;16(1):15–9. https://doi.org/10.1177/1524839914555887 .

Pepall E, James R, Earnest J. Guidelines for conducting rapid participatory appraisals of community health needs in developing countries: experience from Tulikup, Bali. Asia Pac J Public Health. 2006;18(3):42–8. https://doi.org/10.1177/10105395060180030801 .

Pennel CL, McLeroy KR, Burdine JN, Matarrita-Cascante D, Wang J. Community health needs assessment: potential for population health improvement. Popul Health Manag. 2016;19(3):178–86. https://doi.org/10.1089/pop.2015.0075 .

Solet D, Ciske S, Gaonkar R, Horsley K, McNees M, Nandi P, et al. Effective community health assessments in King County, Washington. J Public Health Manag Pract. 2009;15(1):33–40. https://doi.org/10.1097/PHH.0b013e3181903c11 .

Spice C, Snyder K. Reviewing self-reported impacts of community health assessment in local health jurisdictions. J Public Health Manag Pract. 2009;15(1):18–23. https://doi.org/10.1097/PHH.0b013e3181903c57 .

Diaz HJ-M, Ainsworth D, Schmidtlein MC. Funding priorities: data-driven approach for prioritizing community health needs in vulnerable communities. Health Promot Pract. 2019;20(4):616–23. https://doi.org/10.1177/1524839918771977 .

Franz B, Cronin CE, Singh S. Are nonprofit hospitals addressing the most critical community health needs that they identify in their community health needs assessments? J Public Health Manag Pract. 2021;27(1):80–7. https://doi.org/10.1097/phh.0000000000001034 .

Cramer GR, Singh SR, Flaherty S, Young GJ. The progress of US hospitals in addressing community health needs. Am J Public Health. 2017;107(2):255–61. https://doi.org/10.2105/AJPH.2016.303570 .

Kazda MJ, Beel ER, Villegas D, Martinez JG, Patel N, Migala W. Methodological complexities and the use of GIS in conducting a community needs assessment of a large US municipality. J Community Health. 2009;34(3):210–5. https://doi.org/10.1007/s10900-008-9143-3 .

Pennel CL, McLeroy KR, Burdine JN, Matarrita-Cascante D. Nonprofit hospitals’ approach to community health needs assessment. Am J Public Health. 2015;105(3):e103–13. https://doi.org/10.2105/ajph.2014.302286 .

Vance RA, Basta TB, Bute JJ, Denham SA. Identifying the health needs in rural Appalachian Ohio: Outcomes of a rural community-academic partnership. Am J Health Sci. 2012;3(2):115–24. https://doi.org/10.19030/ajhs.v3i2.6942 .

Bias T, Abildso C, Sarkees E. The Importance of Individual-Site and System-Wide Community Health Needs Assessments. Front Public Health. 2020;20. https://doi.org/10.3389/fpubh.2020.00020 .

Billings JR, Cowley S. Approaches to community needs assessment: a literature review. J Adv Nurs. 1995;22(4):721–30. https://doi.org/10.1046/j.1365-2648.1995.22040721.x .

Brazil K, Anderson M. Assessing health service needs: Tools for health planning. HealthC Manage Forum. 1996:22–7. https://doi.org/10.1016/S0840-4704(10)60940-4 .

Campbell S. Assessing community healthcare needs: lessons from Africa. Nurs Stand. 2001;15(47):41. https://doi.org/10.7748/ns2001.08.15.47.41.c3070 .

Carlton EL, Singh SR. Joint community health needs assessments as a path for coordinating community-wide health improvement efforts between hospitals and local health departments. Am J Public Health. 2018;108(5):676–82. https://doi.org/10.2105/ajph.2018.304339 .

Declercq ER, Bichell TJV, Center JK. Population-based needs assessment: bringing public health to midwifery practice. J Nurse Midwifery. 1997;42(6):478–88. https://doi.org/10.1016/s0091-2182(97)00081-5 .

Evans-Agnew R, Reyes D, Primomo J, Meyer K, Matlock-Hightower C. Community health needs assessments: Expanding the boundaries of nursing education in population health. Public Health Nurs. 2017;34(1):69–77. https://doi.org/10.1111/phn.12298 .

Guareschi PA, Jovchelovitch S. Participation, health and the development of community resources in Southern Brazil. J Health Psychol. 2004;9(2):311–22. https://doi.org/10.1177/1359105304040896 .

Holakouie NK, Ahmadvand A, Ahmadnezhad E, Alami A. A community assessment model appropriate for the Iranian community. Iran J Public Health. 2014;43(3):323.

Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health. 1998;19(1):173–202. https://doi.org/10.1146/annurev.publhealth.19.1.173 .

King CJ, Roach JL. Community health needs assessments: a framework for America’s hospitals. Popul Health Manag. 2016;19(2):78–80. https://doi.org/10.1089/pop.2015.0053 .

Misra R, Ballard D. Community needs and strengths assessments as an active learning project. J Sch Health. 2003;73(7):269–71. https://doi.org/10.1111/j.1746-1561.2003.tb06577.x .

Powell RE, Doty AM, Rising KL, Karp DN, Baehr A, Carr BG. A content analysis of nonprofit hospital community health needs assessments and community benefit implementation strategies in Philadelphia. J Public Health Manag Pract. 2018;24(4):326–34. https://doi.org/10.1097/phh.0000000000000621 .

Reece SM. Community analysis for health planning: strategies for primary care practitioners. The. Nurse Pract. 1998;23(10):46 49, 53-46 passim.

Stoto MA, Straus SG, Bohn C, Irani P. A web-based tool for assessing and improving the usefulness of community health assessments. J Public Health Manag Pract. 2009;15(1):10–7. https://doi.org/10.1097/01.PHH.0000342944.48703.21 .

Wetta RE, Pezzino G, LaClair B, Orr S, Brown MB. Voices across Kansas: community health assessment and improvement efforts among local health departments. J Public Health Manag Pract. 2014;20(1):39–42. https://doi.org/10.1097/phh.0b013e3182a9572a .

Download references

Acknowledgements

We acknowledge contribution of the research assistants helped with data extraction.

This work was funded by department of UHC Life course/Integrated Health Services (IHS), World Health Organization (WHO) headquarter (HQ). ZA received the research grant. The authors HR, AS, and SE from WHO commissioned the study, contributed to the direction of the work, and commented on the drafts.

Author information

Authors and affiliations.

Department of Universal Health Coverage/Health Systems (UHS), World Health Organization, Regional Office for the Eastern Mediterranean, Cairo, Egypt

Hamid Ravaghi

Department of Integrated Health Services (IHS), World Health Organization, Headquarters, Geneva, Switzerland

Ann-Lise Guisset

Department of Healthier Populations (DHP), World Health Organization, Regional Office of Eastern Mediterranean Region, Cairo, Egypt

Samar Elfeky

Center for Tropical Medicine and Global Health, Nuffield Department of Medicine, University of Oxford, Oxford, UK

Naima Nasir

National library and Archives of Iran, Tehran, Iran

Sedigheh Khani

National Institute of Health Research (NIHR), Tehran University of Medical Sciences, Tehran (TUMS), Tehran, Iran

Elham Ahmadnezhad

Zhaleh Abdi

You can also search for this author in PubMed   Google Scholar

Contributions

AG, HR and SE conceived the study and participated in its design. SK conducted the literature search and prepared the search results for analysis. NN developed the study framework, the data abstraction forms and the manuscript outline. The literature was analysed by ZA, EA and NN under the supervision of HR and AG. ZA drafted the final version of the manuscript and HR, NN, AG and SE reviewed it. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Zhaleh Abdi .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors have no competing interests. The content of this manuscript is solely the responsibility of the authors and does not necessarily represent the official views of the World Health Organization (WHO).

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

PubMed database search strategy.

Additional file 2.

Content of the extraction forms.

Additional file 3.

List of included empirical papers [ 156 – 159 ].

Additional file 4.

List of included non-empirical papers [ 160 -– 175 ] .

Additional file 5.

Health indicators collected by community health assessment surveys.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Ravaghi, H., Guisset, AL., Elfeky, S. et al. A scoping review of community health needs and assets assessment: concepts, rationale, tools and uses. BMC Health Serv Res 23 , 44 (2023). https://doi.org/10.1186/s12913-022-08983-3

Download citation

Received : 20 May 2022

Accepted : 19 December 2022

Published : 17 January 2023

DOI : https://doi.org/10.1186/s12913-022-08983-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health Needs Assessment
• Needs Assessment
• Assets Assessment
• Population Health
• Scoping Review

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Developing national cancer survivorship standards to inform quality of care in the United States using a consensus approach

• Open access
• Published: 13 May 2024

Cite this article

You have full access to this open access article

• Michelle A. Mollica 1 ,
• Gina McWhirter 2 ,
• Emily Tonorezos 1 ,
• Joshua Fenderson 3 , 4 ,
• David R. Freyer 5 , 6 ,
• Michael Jefford 7 , 8 , 9 ,
• Christopher J. Luevano 10 ,
• Timothy Mullett 11 ,
• Shelley Fuld Nasso 12 ,
• Ethan Schilling 13 ,
• Vida Almario Passero 14 , 15 , 16 &

the National Cancer Survivorship Standards Subject Matter Expert Group

717 Accesses

23 Altmetric

Explore all metrics

To develop United States (US) standards for survivorship care that informs (1) essential health system policy and process components and (2) evaluation of the quality of survivorship care.

The National Cancer Institute and the Department of Veterans Affairs led a review to identify indicators of quality cancer survivorship care in the domains of health system policy, process, and evaluation/assessment. A series of three virtual consensus meetings with survivorship care and research experts and advocates was conducted to rate the importance of the indicators and refine the top indicators. The final set of standards was developed, including ten indicators in each domain.

Prioritized items were survivor-focused, including processes to both assess and manage physical, psychological, and social issues, and evaluation of patient outcomes and experiences. Specific indicators focused on developing a business model for sustaining survivorship care and collecting relevant business metrics (e.g., healthcare utilization, downstream revenue) to show value of survivorship care to health systems.

Conclusions

The National Standards for Cancer Survivorship Care can be used by health systems to guide development of new survivorship care programs or services or to assess alignment and enhance services in existing survivorship programs. Given the variety of settings providing care to survivors, it is necessary for health systems to adapt these standards based on factors including age-specific needs, cancer types, treatments received, and health system resources.

Implications for Cancer Survivors

With over 18 million cancer survivors in the United States, many of whom experience varied symptoms and unmet needs, it is essential for health systems to have a comprehensive strategy to provide ongoing care. The US National Standards for Survivorship Care should serve as a blueprint for what survivors and their families can anticipate after a cancer diagnosis to address their needs.

Similar content being viewed by others

Scoping review of patient-centered care approaches in healthcare

Survivorship care for people affected by advanced or metastatic cancer: MASCC-ASCO standards and practice recommendations

Access, acceptance and adherence to cancer prehabilitation: a mixed-methods systematic review

Avoid common mistakes on your manuscript.

Introduction

A cancer survivor is any individual from the point of diagnosis through the balance of life [ 1 ]. There are over 18 million cancer survivors in the United States [ 2 ], and with advances in diagnostic and treatment capabilities and the aging population, this number is expected to grow. People with cancer have unique survivorship needs, including physical and psychological symptoms both during and after their treatment, risk for recurrence and subsequent cancers, and social needs. As a result, most survivors require long-term follow-up care.

Survivorship care is multifaceted, and recommendations have included surveillance for recurrence and new cancers, prevention and management of physical and psychosocial symptoms, and promoting healthy behaviors [ 3 , 4 ]. While survivorship guidelines exist [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ], the delivery of survivorship care, including what care is delivered, to whom it is delivered, and who delivers the care, varies greatly based on factors including care setting, geographical area, and personal resources. Survivorship care is often fragmented, depending on survivors to seek care from multiple providers without a coordinated system. This is further exacerbated by differing philosophies concerning when survivorship care should be delivered (e.g., post-treatment for those treated with curative intent versus post-diagnosis for anyone with a cancer diagnosis). Survivorship care for many people in the United States is suboptimal, leaving survivors with persistent symptoms, unmet needs, and lack of access to comprehensive care.

There have been several previous efforts to define survivorship care. The LIVESTRONG Essential Elements of Survivorship Care were developed in 2011 with the goal of building consensus in the survivorship community around how best to address the needs of post-treatment survivors [ 14 ]. The American College of Surgeons’ Commission on Cancer (CoC) Survivorship Standard 4.8, recently updated in 2019, defined requirements for CoC accredited programs [ 15 ]. The updated survivorship standards require a survivorship coordinator, a survivorship program documenting a minimum of three services offered each year to support patients, and a focus on enhancing existing and developing new services. This revised standard was an update from the 2016 survivorship standard that required documentation of a survivorship care plan for patients with early-stage cancer treated with curative intent [ 16 ]. In addition, the Quality of Survivorship Care Framework was developed to define the key components of quality survivorship care that are applicable to diverse populations of adult cancer survivors and was intended to inform clinical care, research, and policy [ 3 ].

Given that people with cancer are treated in diverse settings, including cancer centers, academic medical centers, and community sites, there is a need for developing a comprehensive set of national standards for health systems to provide quality survivorship care. The overall goal of this project was to build upon existing efforts to develop national standards for survivorship care that can be utilized by all healthcare systems to assess the quality of existing survivorship care and guide the development of new programs and services. Standards of care represent recommendations for health systems that apply to the patients they serve. Specifically, we sought to define standards for (1) essential health system policy and process components of survivorship care programs and (2) the evaluation of the quality of survivorship care.

The Biden Cancer Moonshot, President Biden’s whole of government response to accelerate progress against cancer and end cancer as we know it, established a goal to develop standards for survivorship care. This project was led by the National Cancer Institute and United States Department of Veterans Affairs, in collaboration with several other Health and Human Services Agencies. Methods were adapted from a previous effort in Australia, where an online modified reactive Delphi survey was completed, followed by a consensus meeting of survivorship experts to inform the Victorian Quality Cancer Survivorship Framework [ 17 ].

Key definitions

For the purpose of this project, we defined a cancer survivor as any individual from the time of diagnosis through the balance of life, diagnosed at any age or stage. We also adapted definitions from Lisy et al. for health system policy, health system process, and evaluation/assessment [ 17 ]. Health system policies were defined as principles and procedures guiding an organization’s capacity and structure to provide survivorship care; health system processes were an organization’s capacity to deliver care through its embedded practices and procedures; and evaluation/assessment were how to measure the impacts of survivorship care within an organization.

Identification of possible indicators

A list of potential indicators in the three domains of health system policy, process, and evaluation/assessment were identified through a review of survivorship and cancer-specific guidelines, the CoC survivorship standard [ 15 ], existing survivorship quality frameworks [ 3 , 17 ], US cancer control plans [ 18 ], and relevant literature. These resources were gathered based on the recommendations of the Task Force and subject matter experts.

Subject matter expert consensus meetings

In 2023, three virtual meetings with survivorship subject matter experts were held to prioritize the most important and feasible indicators to include in the standards. The three meetings were iterative and invited subject matter experts included leading national and international experts in clinical survivorship care, survivorship research, implementation science, health policy, and survivor advocates. Subject matter experts were chosen based on their knowledge of the evidence related to survivorship care and/or their experience in providing care, informing health policy, and/or conducting survivorship care delivery research. We utilized a snowball approach to identify experts and accepted additional recommendations from invited experts, with the overall goal of collectively representing diverse perspectives and experiences related to survivorship. A total of 35 experts participated in the meetings. Additionally, these meetings were open for public viewing and attendees were able to submit comments and questions for consideration and comment.

Meeting 1 focused on providing background to the project, an open discussion among the experts, and individual polling where experts rated the importance of each possible survivorship indicator and identified other indicators for consideration in the next round. Importance was defined using the definition from Lisy and colleagues, as “a core component in achieving survivorship care and can be used to measure the quality of survivorship care” [ 17 ]. For the first meeting, experts were asked only to consider the importance of each indicator rather than also considering the feasibility of implementing and collecting this information. Experts could also suggest edits to the indicators. Questions from Meetings 1–3 can be found in the supplementary information (Appendix A ).

Responses from the Meeting 1 poll were aggregated to identify those rated most important and those rated least important. Based on those results and suggestions from experts and public viewers on edits and additional indicators, an updated list of 15–20 indicators in each domain (policy, process, evaluation/assessment) was developed. Meeting 2 was then held one week later, where results from Meeting 1 were shared, including the indicators rated most important and those rated least important. Following was an open discussion of the results among the experts, including a discussion of feasibility. Experts were then asked to select the top 10 most important and feasible (to implement and/or collect) indicators within each domain; they could also suggest edits to the indicators.

Responses from Meeting 2 were then aggregated to identify the top 10 rated most important and feasible indicators in each domain. Results were shared with experts during Meeting 3, followed by an open discussion of the results. A final poll was conducted where experts were asked to suggest edits to the top 10 indicators in domain and to identify indicators that did not make the top 10 but should be considered for inclusion in the final standards.

Based on suggested edits and additions during the Meeting 3 poll and through refinement by the co-chairs, a final set of standards was developed that includes 10 indicators in health system policy, processes, and evaluation/assessment.

Meeting 1 results

The poll for Meeting 1 included 18 indicators for health system policy, 33 indicators for processes, and 20 indicators for evaluation/assessment. Based on polling results, the policy indicators rated highest importance were a policy requiring establishment of a survivorship program, outlining a team of multidisciplinary health professionals included in the survivorship program, collection of data on survivors’ experience of care and patient-reported outcomes, stratifying survivors to appropriate models of care, provision of support services to survivors based on needs, consideration of approach and timing of transitions in survivorship care, training for healthcare providers, and designation of an organizational survivorship care leader. The policy indicators rated lowest importance were a policy for documenting survivorship care reporting requirements to a government agency, public reporting and dissemination of survivorship outcomes, documenting a minimum of three services offered each year to support patients and survivors, and providing access to prescription produce programs using existing systems/programs. The process indicators rated highest importance were assessment of emotional and psychological effects of cancer and its treatment, physical effects during and following cancer treatment, risk of recurrence or new cancers, practical and social effects (e.g., financial challenges), lifestyle behaviors, and provided with treatment, referrals, and advice to manage physical, emotional, and social effects. The process indicators rated lowest importance were providing the opportunity for participating in research including clinical trials, providing support or referrals for other medical or chronic conditions that are non-cancer related, providing access to advice on vaccinations, providing a meeting to plan survivorship care at the time of diagnosis, providing medically tailored food and nutrition services, providing information and access to complementary health services to support overall health and well-being, and providing a consultation with palliative care. For evaluation , indicators rated highest importance were survivors’ patient-reported outcomes, quality of life, patient-reported experiences of care, return to work, and functional capacity. The evaluation indicators rated lowest importance were overall cost of survivorship care to the health system, number of survivors provided with a survivorship care plan, health professionals’ view of survivorship care, survivors’ hospital admissions, number of referrals made for survivors, and number of primary care providers who are sent a survivorship care plan.

Meeting 2 results

The poll for Meeting 2 included the top 15 indicators for health system policy, 20 indicators for processes, and 20 indicators for evaluation/assessment. The indicators that were ranked in the top 10 for policy included a policy that requires establishment or existence of a survivorship program either on-site or by referral; that describes framework for the provision of survivorship care informed by relevant survivorship guidelines (e.g., ASCO, NCCN, ACS), on stratifying survivors to appropriate models of care; that designates an organizational survivorship care lead who evaluates compliance with standards, has senior role in healthcare system, and includes succession plan for the role, outlining team of multidisciplinary health professionals included in survivorship program; and that considers approach and timing of transitions in survivorship care (e.g., pediatric to adult, acute to primary care, oncology team to survivorship team), for the provision of support services to survivors with special needs and from diverse backgrounds (e.g., navigators, interpreters), for training healthcare providers to deliver survivorship care, for collection of data on survivors’ experience of survivorship care and patient-reported outcomes, and for outlining business case/plan with funding allocated for survivorship care (to include budget). Policy indicators not ranked in the top 10 included outlining the provision of needs assessment tools for survivors at certain time points post-treatment; requiring survivorship-focused information available in other languages or different formats for low-literacy readers; outlining the role of survivors in design, evaluation, and reporting of progress; documenting survivorship care reporting requirements to relevant organizational executive committee; and collecting data on caregivers’ experiences of survivorship care.

For processes , the indicators that were ranked in the top 10 were that cancer survivors were provided access to a survivorship program which addresses the needs of cancer survivors either on-site or by referral; assessed for physical effects during and following cancer treatment, including monitoring for late effects and chronic conditions, and provided with treatment and/or referrals; assessed for emotional and psychological effects of cancer and its treatment and provided with treatment and/or referrals; assessed for practical and social effects of cancer and its treatment (e.g., relationship difficulties, financial challenges, education and employment/return to work) and provided with resources and/or referrals, provided with recommendations regarding surveillance for recurrent or new cancers; assessed for their risk of recurrence or new cancers, including family history and genetic testing; assessed for lifestyle behaviors with recommended management and/or provided with appropriate referral (e.g., smoking cessation, promoting physical activity); provided with access to allied health services (e.g., nutrition, physical therapy, sexual health, fertility services, rehabilitation, dental and podiatry services); provided with access to specialty care services to manage potential late effects (e.g., cardiology); assessed for financial hardship/toxicity and provided with resources and support; and provided with care planning conversations including coordination of care with primary care provider and/or other multidisciplinary health professionals involved in their care. The process indicators not ranked in the top 10 were providing care consistent with their goals, providing access to care to manage fertility and reproductive concerns, providing access to age-specific survivorship care, providing access to primary care services, providing access to age- and gender-appropriate cancer screening or referrals to appropriate screening services, and providing access to tobacco cessation services.

In the domain of evaluation/assessment, indicators ranked in the top 10 were survivors’ and caregivers’ patient-reported outcomes, including quality of life, functional capacity, survival rates (1, 5, and 10 years), experiences of care, return to work, rate of recurrence and new cancers, number and characteristics of survivors lost to follow-up, number of survivors with subsequent chronic condition, rate of survivor service referrals and completions, and relevant business metrics to show return on investment of survivorship care to the healthcare system. Evaluation indicators that were not ranked in the top 10 were collecting data on the number of health professionals trained to provide survivorship care, the number of survivors who have their needs assessed at certain times post-treatment, overall cost of care to survivors and caregivers, survivors’ emergency care and urgent care utilization, number of survivors stratified to different models of care, and oncology providers’ view of the role of nurses and advanced practice providers in survivorship care.

Meeting 3 results

The poll for Meeting 3 included an updated list of the top 10 highest-rated indicators in each domain, as well as a list of the indicators that did not make the top 10. Experts were asked to suggest edits, including suggestions to combine indicators, as well as identify lower-rated indicators to consider for inclusion. Lower-rated indicators that were identified by at least 40% of experts to consider for inclusion were then either combined with other indicators or added to the final set. For health system policy, this included a policy outlining the role of survivors in design, evaluation, and reporting of progress. For processes, this included providing care that is consistent with goals, and consideration of age-specific care. For evaluation/assessment, this included the number of health professionals trained to provide survivorship care.

Final cancer survivorship standards

The National Cancer Standards for Survivorship Care are presented below and in Table  1 and include the top 10 indicators in each of the three domains of health system policy, process, and evaluation/assessment.

Health system policy

The organization has a policy that includes:

Establishment or existence of a survivorship program either on-site, through telehealth, or by referral

A framework for the provision of survivorship care informed by survivor stakeholders and relevant survivorship guidelines (e.g., American Society of Clinical Oncology, National Comprehensive Cancer Network, Children’s Oncology Group)

A description of multidisciplinary care, including each team member’s specific roles and responsibilities and workflow(s) for referrals to team members

An overview of how to stratify and refer survivors to appropriate models of care based on age, treatments, and risk factors

Description of the approach and timing of transitions in survivorship care and shared care (e.g., pediatric to adult providers and settings, oncology team to survivorship team and/or primary care) and efforts to prevent/mitigate loss to follow-up care

An outline for the provision of information for support services (e.g., navigators, social work, interpreters) for survivors based on their needs (including but not limited to health, insurance, and financial literacy, disability status), including survivors from diverse and underserved backgrounds

Identification of an executive-level survivorship care lead (with succession plan) whose role is to ensure compliance with standards, with reporting to an appropriate executive committee

Collection of longitudinal data on survivors’ experience of survivorship care and patient-reported outcomes

Requirements and methods for training healthcare providers (either on-site or through an external training program) to deliver survivorship care within their scope of practice

A business case/plan, including budget, with funding allocated for survivorship care

Health system processes

Cancer survivors are…

Provided with access and referral to a survivorship program that addresses the needs of cancer survivors either on-site, through telehealth, or by referral

Assessed at multiple points in their follow-up care for physical effects during and following cancer treatment, including monitoring for late effects and chronic conditions, and provided with treatment and/or referrals

Provided with access to appropriate specialty care services to manage potential late effects (e.g., cardiovascular issues) either on-site, through telehealth, or by referral

Assessed at multiple points in their follow-up care for emotional and psychological effects of cancer and its treatment and provided with treatment and/or referrals

Assessed for practical and social effects of cancer and its treatment (e.g., social risks, health-related social needs, education and employment/return to work or school) and provided with resources and/or referrals

Assessed for their risk of recurrence or new cancers, including family history and genetic testing, and provided with recommendations and referrals regarding surveillance for recurrence or new cancers

Assessed for lifestyle behaviors and provided with recommended strategies for management and appropriate referrals or education as needed (e.g., smoking cessation, diet/nutrition counseling, promoting physical activity)

Provided with access and referrals to appropriate supportive health services (e.g., nutrition, occupational and physical therapy, rehabilitation, sexual health, fertility services, dental and podiatry services)

Assessed for financial hardship/toxicity and concerns regarding insurance coverage, and provided with resources and support as needed

Engaged in the care planning process including discussion of shared goals of care, advanced care planning, and coordination of care with providers and services (e.g., primary care provider, other health professionals, and community-based services) as needed

Health system evaluation/assessment

The organization has a process to collect data on…

Survivors’ patient-reported outcomes, including quality of life, and experiences of survivorship care

Survivors’ functional capacity

Survivors' return to previous participation in paid and unpaid work/ school/ productive activities of living

Survival rates (1, 5, and 10 years) from the time of diagnosis

Rate of recurrence

Rate of subsequent cancers

Number and relevant characteristics (demographics, clinical factors) of survivors lost to follow-up

Caregivers’ experiences and unmet needs

Number of health professionals trained to provide survivorship care

Relevant business metrics to show return on investment of survivorship care to the healthcare system (e.g., healthcare utilization, rate of referrals and completion, downstream revenue)

Efforts to advance survivorship care have largely been focused on development of evidence-based guidelines and defining the key components of quality survivorship care. Survivorship care services vary greatly among cancer centers and in the community [ 19 ]. Given the growing population of survivors treated in a variety of care settings, it is essential to define a standard for health systems to care for survivors. This current effort aimed to address this gap by developing national standards to define and prioritize key health system policy, process, and evaluation/assessment indicators. While evidence-based guidelines inform provider practices [ 5 ], and the Nekhlyudov framework identifies key components to survivorship care [ 3 ], the standards presented herein build on this previous work. They are intended to be utilized to assess survivorship programs within a health system or organization to address the comprehensive needs of cancer survivors during and after treatment.

The methods for this project were adapted from the Victorian Quality Cancer Survivorship Framework [ 17 ], and the resulting indicators differed in several ways. First, consensus meeting discussions included the need to de-emphasize survivorship care plan documentation, given limited evidence on improving survivor outcomes [ 20 ]. In addition, these standards incorporate all modalities to offer survivorship care services, including telehealth. Experts also identified the need to emphasize support for care transitions across the continuum from diagnosis forward, to include a policy requiring training of healthcare professionals to deliver survivorship care, and subsequently to assess the number of providers trained. Though it was recognized that assessment of survival would be challenging, experts also recognized the need to include and aspire to collect long-term survival data after diagnosis with cancer (1, 5, and 10 years). Finally, US standards include a policy to develop a business case/plan with funding allocated for survivorship care, as well as relevant business metrics to show return on investment for survivorship care. Experts in the consensus meeting stressed the need for a sustainable business model for delivering survivorship care services that is evaluated longitudinally using appropriate metrics including (but not limited to) overall healthcare utilization, rate of referrals and completion, and downstream revenue to the organization or healthcare system. If organizations are to provide quality care for the growing number of survivors, it will be critical to show financial impacts for the healthcare system. An additional process indicator focused specifically on assessing and mitigating survivors’ financial hardship and concerns regarding insurance coverage. This is not surprising given the high proportion of cancer survivors who reporting experiencing financial challenges in the United States [ 21 ].

Consensus meetings also discussed considerations that health systems should take when implementing these standards. In the area of health system policy, experts and advocates noted that these indicators could be combined into one survivorship policy that informs care system-wide. One advantage of separating these indicators, however, is that key informants or stakeholders could be included in development or writing the individual policies. Furthermore, the impact of separate policy changes could be evaluated individually. Additionally, several experts noted that process indicators related to physical, psychological, and social impacts of cancer and its treatment should go beyond only assessment for late effects and should include management and specialty referral, as indicated. Experts and advocates also noted that in the area of evaluation/assessment, validated measures should be utilized whenever possible. While this process was not intended to endorse specific measures or tools, participants emphasized an expectation that validated, patient-centered measures would be used. Finally, health systems should ensure consent has been obtained from survivors and caregivers before assessment.

The final set of standards represents input from survivorship experts and advocates and can be implemented in a variety of settings. A key next step in this work is to implement the standards within healthcare systems that are developing new survivorship programs or have existing programs or services. Healthcare systems that provide care for people after a cancer diagnosis, including but not limited to cancer centers, may use these standards to assess organizational alignment and enhance their survivorship care services. After aligning with the standards, there will be a need to evaluate for feasibility, potential for sustainability, and impact on survivor outcomes. It is important to note that use of these standards by health systems is voluntary, and components of care may or may not be covered by public or private health insurance.

There are several key considerations when implementing these survivorship standards to inform survivorship program development or assess alignment with the indicators. First, while these standards were intended to inform health systems caring for cancer survivors diagnosed at any age, with any cancer, and at any stage, there is a need to tailor care services based on specific factors, including age, setting, and specific cancer types and treatment. For example, while we utilized the NCI recognized definition of a cancer survivor from the time of diagnosis through the balance of life [ 1 ], the standards could also be applied to post-treatment survivors. In addition, survivors of pediatric cancers diagnosed between birth and 15 years may have markedly different needs from survivors of adolescent and young adult (AYA) or older adult survivors [ 22 , 23 ]. For patients diagnosed as young children, survivorship care and research are already highly developed with effective, existing clinical models that constitute “standard of care” for this population. Indeed, in many ways pediatric survivorship care and research have inspired adult-focused efforts. But in pediatric cancer survivorship, particularly for well-established treatments, the evidence base for late effects and their trajectory is well-established and comprehensive. For most childhood cancer survivors, recommended survivorship care for late effects monitoring and management is generally annual, lifelong follow-up, including transition to adult-focused care during young adulthood [ 24 ]. There is still significant work to be done to improve transition services and outcomes [ 25 ], which is one area where the standards could be very informative. Existing guidelines for childhood cancer survivorship care should continue to be utilized [ 13 ].

For survivors of AYA cancers (diagnosed between 15 and 39 years old), health systems and providers must pay close attention to the unique needs of this population. While the components of their survivorship care may be similar to older adults, AYAs are particularly vulnerable to adverse impacts of cancer on education, career development, work, financial status, and psychosocial needs. Fertility is a particular medical concern for this cancer population. Although a separate, parallel set of survivorship standards for survivors of AYA cancer may not be necessary, it is crucial these standards be applied in a manner responsive to their needs. For older adult survivors, geriatric assessment and focused provider training could be incorporated to address the unique considerations of older adults with a history of cancer [ 26 ]. Overall, the standards should be used as a guide for health systems to adapt based on the known needs of populations served.

Though the results of this work represent national standards for survivorship care, these standards can also be utilized to inform survivorship research. The NCI has supported key efforts in survivorship care, including funding opportunities focused on addressing primary care for cancer survivors [ 27 ] and optimizing survivorship care for survivors transitioning between oncology and non-oncology providers [ 28 ]. A challenge in delivering and evaluating survivorship care, however, is that there has not been an accepted national standard. This project represents consensus agreement of national experts on essential policy, process, and evaluation components to survivorship care that health systems should utilize based on the available evidence. Rigorous evaluation of the implementation and outcomes of these standards will be critical to show continued value to follow-up care for people after a cancer diagnosis [ 29 ]. Additionally, the evaluation/assessment indicators of these standards may be used as meaningful endpoints for survivorship care interventions to show impact on survivor and health system outcomes.

These standards represent a key foundation for improving the delivery of survivorship care across the United States; however, there are limitations to this work. First, though a robust review was conducted to identify potential indicators, it is possible that specific literature may have been inadvertently missed. In addition, the consensus meetings represented a diverse panel of experts and survivor advocates who provided feedback and input. It is possible, however, that some perspectives were not represented in the expert group. One important perspective missing is that of a healthcare business administrator, who will be essential in converting these standards to implementation of survivorship programs. A key next step in this work will be collaboration with healthcare administrators and payors to translate these recommendations into action. The selection of experts also may lead to limitations in the prioritization of indicators. Future empirical support is needed to provide evidence of the outcomes of implementing the standards.

Data availability

No datasets were generated or analyzed during the current study.

This study has not been previously presented.

National Cancer Institute. Office of cancer survivorship: definitions. 2020; Available from: https://cancercontrol.cancer.gov/ocs/statistics#definitions .

Miller, K.D., et al. Cancer treatment and survivorship statistics, 2022. CA Cancer J Clin. 2022;72(5):409–436. https://doi.org/10.3322/caac.21731 .

Nekhlyudov L, et al. Developing a quality of cancer survivorship care framework: implications for clinical care, research and policy. J Natl Cancer Inst. 2019;111(11):1120–30.

Article   PubMed   PubMed Central   Google Scholar  

From Cancer Patient to Cancer Survivor: Lost in Transition. Washington, DC: The National Academies Press. 2006.  https://doi.org/10.17226/11468 .

Sanft T, et al. NCCN Guidelines® Insights: survivorship, version 1.2023. J Natl Compr Canc Netw. 2023;21(8):792–803.

Article   PubMed   Google Scholar  

Cohen EE, et al. American Cancer Society head and neck cancer survivorship care guideline. CA Cancer J Clin. 2016;66(3):203–39.

Nekhlyudov L, et al. Head and Neck cancer survivorship care guideline: American Society of Clinical Oncology clinical practice guideline endorsement of the American Cancer Society Guideline. J Clin Oncol. 2017;35(14):1606–21.

Skolarus T, et al. American Cancer Society prostate cancer survivorship care guidelines. CA: A Cancer Journal for Clinicians. 2014;64:225–49.

PubMed   Google Scholar  

El-Shami K, et al. American Cancer Society colorectal cancer survivorship care guidelines. CA Cancer J Clin. 2015;65(6):428–55.

Runowicz C, et al. American Cancer Society/American Society of Clinical Oncology breast cancer survivorship care guideline. J Clin Oncol. 2016;34:611–35.

Article   CAS   PubMed   Google Scholar  

American Society of Clinical Oncology, American Society of Clinical Oncology clinical practice survivorship guidelines and adaptations: summary of recommendations. Alexandria. VA: American Society of Clinical Oncology; 2014.

Google Scholar  

Children’s Oncology Group, Long-term follow-up guidelines for survivors of childhood, adolescent, and young adult cancer. 2013, Children’s Oncology Group: Monrovia, CA.

Children’s Oncology Group. Long-term follow-up guidelines for survivors of childhood, adolescent, and young adult cancers Version 6.0. 2023 [cited 2024 January 30]; Available from: http://www.survivorshipguidelines.org/ .

Rechis, R., et al. The essential elements of survivorship care: a LIVESTRONG brief. 2011 [cited 2018; Available from: https://www.livestrong.org/sites/default/files/what-we-do/reports/EssentialElementsBrief.pdf .

Blaes AH, et al. Survivorship care plans and the commission on cancer standards: the increasing need for better strategies to improve the outcome for survivors of cancer. JCO Oncology Practice. 2020;16(8):447–50.

Shulman LN, et al. Treatment summaries and survivorship care plans: the approach by the commission on cancer to increase use. J Oncol Pract. 2015;11(1):40–1.

Lisy, K., et al. How do we define and measure optimal care for cancer survivors? An online modified reactive Delphi study. Cancers (Basel), 2021;13(10).

Mollica MA, et al. Survivorship objectives in comprehensive cancer control plans: a systematic review. J Cancer Surviv. 2020;14(2):235–43.

Anampa-Guzmán, A., et al. Online description of services provided in adult survivorship programs across U.S. accredited cancer centers . J Cancer Surviv. 2024;18(1):79–83. https://doi.org/10.1007/s11764-023-01361-w .

Jacobsen PB, et al. Systematic review of the impact of cancer survivorship care plans on health outcomes and health care delivery. J Clin Oncol. 2018;36(20):2088–100.

Abrams HR, et al. Financial toxicity in cancer care: origins, impact, and solutions. Translational Behavioral Medicine. 2021;11(11):2043–54.

Siembida EJ, et al. Measuring health-related quality of life in adolescent and young adult cancer survivors with the National Institutes of Health Patient-Reported Outcomes Measurement Information System(®): comparing adolescent, emerging adult, and young adult survivor perspectives. Psychooncology. 2021;30(3):303–11.

Anderson C, et al. Noncancer mortality among adolescents and young adults with cancer. Cancer. 2019;125(12):2107–14.

Tonorezos ES, et al. Models of care for survivors of childhood cancer from across the globe: advancing survivorship care in the next decade. J Clin Oncol. 2018;36(21):2223–30.

Marchak JG, et al. Transition practices for survivors of childhood cancer: a report from the Children’s Oncology Group. J Cancer Surviv. 2023;17(2):342–50.

Dale W, et al. How is geriatric assessment used in clinical practice for older adults with cancer? A survey of cancer providers by the American Society of Clinical Oncology. JCO Oncol Pract. 2021;17(6):336–44.

National Cancer Institute. RFA-CA-23–025: addressing the primary care needs of cancer survivors (U01 clinical trial required). 2023 [cited 2024 January 19]; Available from: https://grants.nih.gov/grants/guide/rfa-files/rfa-ca-23-025.html .

National Cancer Institute. RFA-CA-19–035: Optimizing the management and outcomes for cancer survivors transitioning to follow-up care. 2020 [cited 2024 January 19]; Available from: https://cancercontrol.cancer.gov/research-emphasis/contribution-to-the-cancer-moonshot/optimizing-management-outcomes .

Halpern M, et al. Myths and presumptions about cancer survivorship. J Clin Oncol. 2024;42(2):134–9.

Download references

Acknowledgements

National Survivorship Standards Subject Matter Expert Group

Open access funding provided by the National Institutes of Health.

Author information

Authors and affiliations.

Division of Cancer Control and Population Sciences, National Cancer Institute, 9609 Medical Center Drive, MSC 9712, Room 3E440, Bethesda, MD, 20892-9762, USA

Michelle A. Mollica & Emily Tonorezos

Department of Veterans Affairs, National Oncology Program, Washington, DC, USA

Gina McWhirter

Hematology/Oncology Service, Brooke Army Medical Center, Defense Health Agency, San Antonio, TX, USA

Joshua Fenderson

Uniformed Services University of Health Sciences, Bethesda, MD, USA

Keck School of Medicine, University of Southern California, Los Angeles, CA, USA

David R. Freyer

Children’s Hospital Los Angeles and USC Norris Comprehensive Cancer Center, Los Angeles, CA, USA

Department of Health Services Research, Peter MacCallum Cancer Centre, Melbourne, VIC, Australia

Michael Jefford

Australian Cancer Survivorship Centre, Peter MacCallum Cancer Centre, Melbourne, VIC, Australia

Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, VIC, Australia

Office of The Assistant Secretary of Defense for Health Affairs, Department of Defense, Washington, DC, USA

Christopher J. Luevano

Department of Surgery, College of Medicine, University of Kentucky, Lexington, KY, USA

Timothy Mullett

National Coalition for Cancer Survivorship, Silver Spring, MD, USA

Shelley Fuld Nasso

Cancer Survivorship Advocate, Carolina Pediatric Therapy, Asheville, NC, USA

Ethan Schilling

Division of Hematology/Oncology, University of Pittsburgh School of Medicine, Pittsburgh, PA, USA

Vida Almario Passero

Section of Hematology/Oncology, VA Pittsburgh Healthcare System, Pittsburgh, PA, USA

VA National TeleOncology, Durham, NC, USA

You can also search for this author in PubMed   Google Scholar

• Catherine Alfano
• , Precilla Belin
• , Anne Blaes
• , Hillary Cavanagh
• , Lanie Francis
• , David R. Freyer
• , Danielle Friedman
• , Shelley Fuld Nasso
• , Patricia Ganz
• , Batsheva Honig
• , Shawna Hudson
• , Linda Jacobs
• , Michael Jefford
• , Nancy Keating
• , Anne Kirchoff
• , Michelle Kirschner
• , Ron Kline
• , Jessica MacIntyre
• , Molly Maher
• , Deborah Mayer
• , Janette Merrill
• , Timothy Mullett
• , Larissa Nekhlyudov
• , Frank Penedo
• , Mackenzi Pergolotti
• , Michael Roth
• , Tara Sanft
• , Alyssa Schatz
• , Ethan Schilling
• , Kathryn Schmitz
• , Lisa Schwartz
•  & Emily Tonorezos

Contributions

MAM, GM, ET, and VAP were responsible for research conceptualization. MAM wrote the main manuscript text in collaboration with all authors. All authors reviewed the study design, results, and the manuscript.

Corresponding author

Correspondence to Michelle A. Mollica .

Ethics declarations

Consent to participate.

No human subject participants were included in this project. Subject matter experts implied consent to participate in the consensus process.

Conflict of interest

The authors declare no competing interests.

Disclaimers

The article was prepared as part of some of the authors’ (MAM, ET) official duties as employees of the US Federal Government. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the National Cancer Institute.

The view(s) expressed herein are those of the author(s) and do not reflect the official policy or position of the Defense Health Agency, Department of Defense, or the US Government. Mention of trade names, commercial products, or organizations does not imply endorsement by Department of Defense or the US Government. The views expressed herein are those of the authors and do not reflect the official policy or position of the Department of Veterans Affairs or the US Government.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 25 KB)

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Mollica, M.A., McWhirter, G., Tonorezos, E. et al. Developing national cancer survivorship standards to inform quality of care in the United States using a consensus approach. J Cancer Surviv (2024). https://doi.org/10.1007/s11764-024-01602-6

Download citation

Received : 04 April 2024

Accepted : 11 April 2024

Published : 13 May 2024

DOI : https://doi.org/10.1007/s11764-024-01602-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health system
• Survivorship
• Care quality
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 17 May 2024

Co-designing Entrustable Professional Activities in General Practitioner’s training: a participatory research study

• Vasiliki Andreou   ORCID: orcid.org/0000-0002-6679-0514 1 , 4 ,
• Sanne Peters   ORCID: orcid.org/0000-0001-6235-1752 1 , 2 ,
• Jan Eggermont   ORCID: orcid.org/0000-0002-8497-1159 3 &
• Birgitte Schoenmakers   ORCID: orcid.org/0000-0003-1909-9613 1  

BMC Medical Education volume  24 , Article number:  549 ( 2024 ) Cite this article

Metrics details

In medical education, Entrustable Professional Activities (EPAs) have been gaining momentum for the last decade. Such novel educational interventions necessitate accommodating competing needs, those of curriculum designers, and those of users in practice, in order to be successfully implemented.

We employed a participatory research design, engaging diverse stakeholders in designing an EPA framework. This iterative approach allowed for continuous refinement, shaping a comprehensive blueprint comprising 60 EPAs. Our approach involved two iterative cycles. In the first cycle, we utilized a modified-Delphi methodology with clinical competence committee (CCC) members, asking them whether each EPA should be included. In the second cycle, we used semi-structured interviews with General Practitioner (GP) trainers and trainees to explore their perceptions about the framework and refine it accordingly.

During the first cycle, 14 CCC members agreed that all the 60 EPAs should be included in the framework. Regarding the formulation of each EPAs, 20 comments were given and 16 adaptations were made to enhance clarity. In the second cycle, the semi-structured interviews with trainers and trainees echoed the same findings, emphasizing the need of the EPA framework for improving workplace-based assessment, and its relevance to real-world clinical scenarios. However, trainees and trainers expressed concerns regarding implementation challenges, such as the large number of EPAs to be assessed, and perception of EPAs as potentially high-stakes.

Accommodating competing stakeholders’ needs during the design process can significantly enhance the EPA implementation. Recognizing users as experts in their own experiences empowers them, enabling a priori identification of implementation barriers and potential pitfalls. By embracing a collaborative approach, wherein diverse stakeholders contribute their unique viewpoints, we can only create effective educational interventions to complex assessment challenges.

Peer Review reports

Introduction

In recent years, the landscape of medical education has significantly transformed due to increasing demands of public accountability and changing patient needs. In response to these evolving demands, competency-based medical education (CBME) has emerged. CBME has been gaining popularity in medical education programs [ 1 ]. In a CBME paradigm, medical curricula are structured based on predefined competencies that physicians should have acquired upon completion of the program [ 2 , 3 ]. Despite the theoretical underpinnings of CBME, its implementation has encountered various obstacles [ 4 ]. Particularly, assessing competencies in real clinical environments has been a major barrier in the effective integration of CBME into medical education systems [ 5 ]. Recognizing this challenge, the concept of Entrustable Professional Activities (EPAs) has emerged.

EPAs are essentially tasks or activities that medical professionals should be able to perform competently and independently by the time they complete their training [ 6 , 7 ]. EPAs are used to assess a learner’s ability to integrate and apply the necessary competencies in real-world clinical practice. They necessitate evaluating a learner’s progress and readiness for independent practice by observing their performance in these key professional activities in clinical practice [ 8 ]. The term “entrustable” indicates that, upon graduation or completion of a specific training period, a supervising physician or mentor should be able to entrust a medical graduate with these activities without direct supervision, considering them proficient and safe for the patients to perform these tasks independently [ 9 , 10 ].

Considering the immense potential, integration and implementation of EPAs has gained rapid momentum, across various health professions and medical specialties [ 11 , 12 ]. Despite this progress, a significant gap notably persists, when it comes to accommodating competing needs of curriculum designers and those of users in practice, namely trainers and trainees [ 13 ]. While the promise of EPAs in facilitating CBME is promising, there is lack of comprehensive evidence incorporating users’ perceptions during the design phase [ 8 , 11 , 14 ]. Therefore, the aim of this study was to design an EPA framework for workplace-based assessment by actively involving clinical educators, trainees and trainers throughout the process.

Setting and participants

This study took place in the interuniversity postgraduate General Practitioner’s (GP) Training, Belgium. To standardize GP Training across Flanders, four Flemish universities (KU Leuven, Ghent University, University of Antwerp, and the Flemish Free University of Brussels) collaboratively developed a postgraduate training program. This training program consists of three different training-phases and rotations, spread through three years, two rotations are in a GP practice, while one takes place at a hospital setting.

The GP Training is overseen by the Interuniversity Centre for GP Training (ICGPT). The ICGPT plays a pivotal role in coordinating and managing various aspects of the curriculum. Among its key responsibilities, the ICGPT oversees the allocation of clinical internships, conducts examinations, facilitates regular meetings between trainees and trainers, and maintains trainees’ learning electronic (e-) portfolios.

In 2018, the ICGPT initiated a shift towards CBME. The rationale of CBME was introduced in the curriculum by integrating first the CanMEDS roles. To facilitate this transition, two clinical competence committees (CCCs), comprising medical doctors and clinical educators from the four universities were appointed. These CCCs were tasked with coordinating workplace-based learning, and curriculum and assessment, respectively.

To align the curriculum with the patient needs in primary care, the two CCCs designated and defined ten different care contexts characteristic of primary care (i.e. short-term care, chronic care, emergency care, palliative care, elderly care, care for children, mental healthcare, prevention, gender related care, and practice management). Subsequently, in 2022, we initiated the process of designing specific EPAs for the GP Training. The EPAs aimed to facilitate and improve workplace-based assessment. These two CCCs participated in the design process, while trainers and trainees were invited to share their opinion as well.

Designing the EPA framework

The design of the EPA framework was based on participatory research design to engage different stakeholders [ 15 ]. Participatory research design is a community-based methodology aiming to create solutions for and with the people who are involved [ 15 ]. This iterative research approach encompassed three fundamental design-stages in a circular relationship, namely design, evaluation and refinement (Fig.  1 ). We executed two distinct iterative cycles, each with a specific group of stakeholders (Fig.  2 ). In cycle 1, we focused on CCCs, fostering discussions and validating the framework. In cycle 2, we involved clinical trainers and trainees, ensuring cross-validation. In the following section, we describe each iterative cycle, indicated as cycle 1 and as cycle 2, respectively.

Three design phases for designing the EPA framework

Process for developing the EPA framework based on participatory design research

In cycle 1, after reviewing relevant literature, we developed a blueprint of 60 EPAs corresponding to the ten different care contexts, already integrated in the curriculum [ 9 , 10 ]. By doing so, we wanted to ensure practical applicability and relevance of our framework within the established educational environment. Afterwards, we linked all EPAs to the CanMEDS competency framework [ 16 ]. We defined competencies as broad statements that describe knowledge, skills and attitudes that GP trainees should achieve during the different training phases [ 17 ]. The CanMEDS framework identifies and describes different competencies for patient-centred care, and comprises seven different roles: medical expert, communicator, collaborator, leader, health advocate, scholar, and professional. By linking EPAs to CanMEDS, we constructed a matrix that served as a structured guide for integrating the EPAs in the workplace. Also, together with the CCCs we defined behavioural and cognitive criteria to anchor entrustment levels [ 9 ]. These criteria described required knowledge, skills, and attitudes in order for an EPA to be entrusted.

In cycle 2, we aimed at operationalising the EPAs, cross validating them by interviewing trainers and trainees, and deciding entrustment levels. Specifically, to operationalise the EPAs, we developed an assessment form, called Clinical Practice Feedback form (Fig.  3 ). We chose to link EPA assessments not only to direct and video observations, but also for case-based discussions. Additionally, we agreed upon entrustment levels and the entrustability scale. Entrustment was anchored on criteria that were defined along the EPAs. We decided to use the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) for validity and reliability reasons (Fig.  4 ) [ 18 ]. The Ottawa scale requires assessors to describe how much supervision they provided to trainees while performing a specific EPA. Concretely, the scale comprises five levels of performance ranging from trainers taking over the activity to trainees performing the activity without supervision (Fig.  3 ) [ 18 ].

Example of Clinical Practice Feedback form available in the e-portfolio

Five levels of entrustment based on the O-SCORE scale [ 19 ]

Data collection and analysis

In cycle 1, we evaluated the EPA blueprint by employing a modified Delphi methodology, with two rounds [ 19 ]. We invited members of the two CCCs ( N  = 14) to give feedback on the EPA blueprint via e-mail and during meetings, scheduled by the ICGPT. Members were asked whether they thought each EPA was necessary for workplace-based assessment and needed to be included in the framework. They were also encouraged to give feedback regarding the formulation of the EPAs. Once we gathered all the comments, we refined the blueprint and sent it back to the CCC members. In cycle 2, we interviewed two trainers and two trainees using semi-structured interviews and following the ‘think-aloud protocol’ [ 20 , 21 , 22 ], where we asked them whether each EPA was necessary and whether they were comprehensible for workplace-based assessment. Participants were required to articulate their thoughts while reading the EPA framework. This enabled us to gain insights into their thought processes and perspectives [ 22 ].

Data collection took place from February 2022 until September 2022. For quantitative data analysis we calculated descriptive statistics of consensus rates using SPSS 27 (IBM SPSS Statistics 27). We analysed qualitative data from CCCs members using content analysis on Microsoft Excel. For analysing data from the interviews with the trainers and trainees, we first verbatim transcribed the interviews, and, then, analysed the data using thematic analysis in NVivo (QSR International) [ 23 , 24 ]. Qualitative data were analysed by two researchers separately to achieve triangulation, while a third researcher was consulted, when discrepancies arose [ 25 ].

Reflexivity and research team

The research team was composed of members with different backgrounds. Two members had a background in education, while the other two members had a background in biomedical sciences and general practice. All authors had research training and experience in medical education research. Methodological and design decisions were in line with the available literature. We predefined methodological steps before commencing the study. To ensure adherence to our design stages, we maintained a detailed logbook to document systematically progression and modifications from our initial protocol. We regularly discussed the results to ensure that our interpretations were close to the data.

In cycle 1, fourteen members of the CCCs gave feedback on the list of 60 EPAs. In the first feedback round, all members agreed that all 60 EPAs were required in the framework. Twenty comments were given regarding the formulation of the EPAs and 16 adaptations were made based on the new suggestions. Comments regarding the formulation were about the use of certain words in order to make the framework understandable. In the second feedback round, consensus was reached on the formulation of the EPAs (Table  1 ).

In cycle 2, we interviewed two trainers and two trainees. CCC members, trainers, and trainees agreed that all EPAs should be included in the framework. From these interviews, we identified three themes. Table  2 presents these three themes alongside their subthemes. Necessity of EPAs was the first theme and included shared mindsets about necessity of EPAs in order to improve workplace-based assessment and difficulties with interpreting the CanMEDS roles.

“ The EPAs are better than the CanMEDS. My trainer and I often do not know what we have to assess…He (the trainer) sometimes gives the same feedback for multiple roles .” (trainee 1).

Second theme was about the relevance of EPAs to clinical practice. Users thought that the EPA framework could easily be linked to their clinical work, promoting assessment and feedback opportunities. They agreed that EPAs were understandable and formulated in intuitive language for clinical work.

“ I think that it (the EPA framework) is quite intuitive. I can see a lot of links between the EPAs and my daily practice .” (trainer 2).

I like the (EPA) framework. My trainer and I already discuss some of these (activities) during our weekly feedback session . (trainee 2)

Third theme included challenges in implementation of EPAs, regarding the large number of EPAs, perception of high-stakes assessment within an e-portfolio, and limitations inherent to the current e-portfolio. First, users expressed their concern regarding the large number of EPAs. They indicated that only a limited number might be feasible because of time constraints in the clinical workplace. Also, users thought that due to the large number of EPAs, trainees would “pick and choose” EPAs where they had performed well. Along with limited functionalities of the current e-portfolio, they indicated that EPAs might be used as showcasing performance instead for workplace-based assessment and feedback purposes. Mainly trainees expressed hesitation to document EPAs where they would need further improvement. They perceived the e-portfolio as a tool more suitable for high-stakes assessments rather than for feedback purposes.

“ The list (of EPAs) is quite extensive… I do want to have a nice portfolio, so for sure I will try to include as many as possible. In case something happens (in my curriculum), I want to show how well I have been performing .” (trainee 1).

“ I normally do not include patient cases that went wrong in my portfolio. Because different people have access to it (the e-portfolio).” (trainee 2).

The aim of this study was to design an EPA framework by actively engaging and collaborating with different stakeholders. To be established as a “good” assessment framework, EPAs should be acceptable by the different stakeholders involved in the assessment process, such as curriculum designers, trainees and trainers [ 26 , 27 ]. Incorporating their opinions and understanding their different needs must be integral to the design process. However, literature regarding EPAs design has mainly focused on experts’ opinion, neglecting users in practice [ 8 ].

From our findings, it becomes apparent that direct involvement and communication among diverse stakeholders are crucial for designing a useful for everyone EPA assessment framework. When various groups are involved in developing educational interventions, competing needs can be optimally addressed [ 28 ]. This optimization fosters a cohesive approach, ensuring high applicability rates and effectiveness, when the EPA framework is used in practice. The need for users’ involvement in the development process is currently demonstrated in the most recent EPA literature [ 29 , 30 ]. Users’ involvement promotes common language and expectations, enhancing the clarity and effectiveness of EPA interventions, and, most importantly, empowers the users themselves by acknowledging their perspectives [ 31 ]. Ultimately, trainees and trainers are the ones using the EPA assessment frameworks during daily clinical practice, and are potentially confronted with unforeseen obstacles.

Additionally, users’ involvement in the process can help to identify potential implementation challenges [ 32 , 33 ]. Our findings indicate differences in opinions regarding implementation of EPAs. In contrast to the CCC members, users expressed their concerns about the large number of EPAs included in the framework. They were particularly concerned about how to use sufficiently and adequately EPA assessments, while juggling clinical work. This concern echoes findings from other studies as well, related to the assessment burden [ 34 ]. In particular, when challenges in assessment processes arise in the clinical workplace, assessment is most probably not performed as intended [ 35 ].

Furthermore, our results illustrate tensions between assessment of learning and assessment for learning. Although the EPA assessments aim to better prepare trainees for clinical practice, users suggested that the purpose of the EPAs might not be explicit for everyone. Since EPAs are a form of assessment, they could potentially lead to strategic behaviours of documenting successful EPAs, and, therefore, creating a fragmented idea about trainees’ performance in clinical practice. Additionally, the use of the current e-portfolio for high-stakes assessments only adds to this tension. Especially, trainees were not comfortable with sharing performance evidence for improvement, because they perceived the stakes as high [ 36 ]. The dilemma between learning versus performing has been the Achilles point in workplace-based assessment [ 37 ]. The lines between assessment and feedback seem to be also blurred in EPAs [ 38 , 39 ].

Involving users during the design process can lead not only to early adaptations and refinement of EPAs, but also to better allocation of resources. In order to ensure successful implementation of EPAs, it is essential to recognize the central role of both trainers and trainees. Future research should focus on training programs designed to equip faculty, trainers, and trainees with a profound understanding of EPAs. Users in practice need rigorous training covering EPA principles, assessment techniques, and feedback strategies [ 40 ]. Moreover, fostering a culture of interdisciplinary collaboration among stakeholder groups is imperative. Encouraging review of assessment tools and facilitating the exchange of opinions during designprocesses can significantly enhance the overall quality of EPA frameworks, and, even more broadly, of workplace-based assessment practices.

Although EPAs are a valuable framework for assessing competencies in workplace settings, integrating other assessment tools is crucial to capture the full spectrum of skills needed to meet patient needs. Future research should focus on combining EPAs with other assessment methods, such as simulation-based assessments, either with standardized patients or with virtual reality, that would allow trainees to demonstrate their clinical and interpersonal skills within safe, controlled environments that closely replicate challenging patient scenarios [ 41 ]. Additionally, incorporating multisource feedback and continuous portfolio assessments could offer a comprehensive view of a trainee’s performance across various settings and interactions [ 42 , 43 ]. Together, these methods would enhance the EPA framework, ensuring a comprehensive assessment of all essential competencies that future physicians should acquire.

Limitations

We need to acknowledge several limitations in this study. First, in medical education research, users’ involvement prerequisites a degree of experience with a specific subject. In our study, we involved users in the early design process of the EPA framework. Although we are aware of this limitation, we intentionally and consciously chose a participatory research design. We believe that users are experts in their own experience, and that they hold the knowledge and capabilities to be involved as partners in the development process. Second, our study involved a low number of users due to difficulties in recruitment. This might have led to recruiting participants who are fully engaged in the educational practices of the GP Training. Nevertheless, our findings are rooted in two methodologies, namely a modified Delphi method and semi-structured interviews. Therefore, we used triangulation to verify our results [ 25 ]. Finally, although workshops are mostly commonly in co-design studies [ 44 ], our study coincided with the last COVID-19 lockdown, necessitating adjustments. To cope with these challenges and uncertainties, we opted for methods that were the most feasible for our participants at that moment. Despite these challenges, the contributions from all stakeholders were invaluable, particularly in exploring potential implementation and evaluation issues.

For EPAs to be successful, they need to be acceptable as an assessment framework by different stakeholders’ groups. Accommodation of competing stakeholders’ needs during the design process is crucial for enhancing acceptability and effectiveness during implementation. Our findings highlight the significance of collaborative efforts to design EPAs, emphasizing its potential to empower users, identify implementation barriers, and pinpoint unintended consequences. Through this collaborative approach, wherein diverse stakeholders contribute their perspectives, we can create effective educational solutions to complex assessment challenges.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

General Practitioner

competency-based medical education

Entrustable Professional Activity

Canadian Medical Education Directives for Specialists

Interuniversity Centre for GP Training

clinical competence committee

Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638–45.

Article   Google Scholar  

Iobst WF, Sherbino J, Cate OT, Richardson DL, Dath D, Swing SR, et al. Competency-based medical education in postgraduate medical education. Med Teach. 2010;32(8):651–6.

Frank JR, Snell L, Englander R, Holmboe ES. Implementing competency-based medical education: moving forward. Med Teach. 2017;39(6):568–73.

Nousiainen MT, Caverzagie KJ, Ferguson PC, Frank JR. Implementing competency-based medical education: what changes in curricular structure and processes are needed? Med Teach. 2017;39(6):594–8.

Lockyer J, Carraccio C, Chan M-K, Hart D, Smee S, Touchie C, et al. Core principles of assessment in competency-based medical education. Med Teach. 2017;39(6):609–16.

Ten Cate O, Scheele F. Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad Med. 2007;82(6):542–7.

Carraccio C, Englander R, Gilhooly J, Mink R, Hofkosh D, Barone MA, et al. Building a framework of entrustable professional activities, supported by competencies and milestones, to bridge the educational continuum. Acad Med. 2017;92(3):324–30.

Ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE guide no. 99. Med Teach. 2015;37(11):983–1002.

Ten Cate O, Taylor DR. The recommended description of an entrustable professional activity: AMEE guide no. 140. Med Teach. 2021;43(10):1106–14.

Carraccio C, Martini A, Van Melle E, Schumacher DJ. Identifying core components of EPA implementation: a path to knowing if a complex intervention is being implemented as intended. Acad Med. 2021;96(9):1332–6.

de Graaf J, Bolk M, Dijkstra A, van der Horst M, Hoff RG, Ten Cate O. The implementation of entrustable professional activities in postgraduate medical education in the Netherlands: rationale, process, and current status. Acad Med. 2021;96(7s):S29-35.

Keeley MG, Bray MJ, Bradley EB, Peterson CM, Waggoner-Fountain LA, Gusic ME. Fidelity to best practices in EPA implementation: outcomes supporting use of the core components framework from the University of Virginia entrustable professional activity program. Acad Med. 2022;97(11):1637–42.

St-Onge C, Boileau E, Langevin S, Nguyen LHP, Drescher O, Bergeron L, et al. Stakeholders’ perception on the implementation of developmental progress assessment: using the theoretical domains framework to document behavioral determinants. Adv Health Sci Educ. 2022;27(3):735–59.

Taylor DR, Park YS, Egan R, Chan MK, Karpinski J, Touchie C, et al. EQual, a novel rubric to evaluate entrustable professional activities for quality and structure. Acad Med. 2017;92(11S):S110-117.

Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. Am J Public Health. 2010;100 Suppl 1(Suppl 1):S40-46.

Frank JR, Snell L, Sherbino J. CanMEDS 2015 Physician competency framework. Ottawa: Royal College of Physicians & Surgeons of Canada; 2015.

Harden RM. Learning outcomes and instructional objectives: is there a difference? Med Teach. 2002;24(2):151–5.

Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa surgical competency operating room evaluation (O-SCORE): a tool to assess surgical competence. Acad Med. 2012;87(10):1401–7.

de Villiers MR, de Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Med Teach. 2005;27(7):639–43.

Patton MQ, Fund RECM. Qualitative research & evaluation methods. SAGE Publications; 2002.

Sargeant J. Qualitative research part II: participants, analysis, and quality assurance. J Graduate Med Educ. 2012;4(1):1–3.

Ericsson KA, Simon HA. How to study thinking in everyday life: contrasting think-aloud protocols with descriptions and explanations of thinking. Mind Cult Act. 1998;5(3):178–86.

Lumivero. NVivo (Version 14). 2023. www.lumivero.com .

Krippendorff K. Content analysis: an introduction to its methodology. Sage; 2018.

Carter N, Bryant-Lukosius D, DiCenso A, Blythe J, Neville AJ. The use of triangulation in qualitative research. Oncol Nurs Forum. 2014;41(5):545–7.

Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 conference. Med Teach. 2011;33(3):206–14.

Norcini J, Anderson MB, Bollela V, Burch V, Costa MJ, Duvivier R, et al. 2018 Consensus framework for good assessment. Med Teach. 2018;40(11):1102–9.

Göttgens I, Oertelt-Prigione S. The application of human-centered design approaches in health research and innovation: a narrative review of current practices. JMIR Mhealth Uhealth. 2021;9(12):e28102.

Bonnie LHA, Visser MRM, Bont J, Kramer AWM, van Dijk N. Trainers’ and trainees’ expectations of entrustable professional activities (EPAs) in a primary care training programme. Educ Prim Care. 2019;30(1):13–21.

van Loon KA, Bonnie LHA, van Dijk N, Scheele F. Benefits of EPAs at risk? The influence of the workplace environment on the uptake of EPAs in EPA-based curricula. Perspect Med Educ. 2021;10(4):200–6.

van Loon KA, Scheele F. Improving graduate medical education through faculty empowerment instead of detailed guidelines. Acad Med. 2021;96(2):173.

Peters S, Bussières A, Depreitere B, Vanholle S, Cristens J, Vermandere M, et al. Facilitating guideline implementation in primary health care practices. J Prim Care Community Health. 2020;11:2150132720916263.

Peters S, Sukumar K, Blanchard S, Ramasamy A, Malinowski J, Ginex P, et al. Trends in guideline implementation: an updated scoping review. Implement Sci. 2022;17(1):50.

Szulewski A, Braund H, Dagnone DJ, McEwen L, Dalgarno N, Schultz KW, et al. The assessment burden in competency-based medical education: how programs are adapting. Acad Med. 2023;98(11):1261–7.

Thaler RH. Nudge, not sludge. Science. 2018;361(6401):431.

Schut S, Driessen E, van Tartwijk J, van der Vleuten C, Heeneman S. Stakes in the eye of the beholder: an international study of learners’ perceptions within programmatic assessment. Med Educ. 2018;52(6):654–63.

Watling CJ, Ginsburg S. Assessment, feedback and the alchemy of learning. Med Educ. 2019;53(1):76–85.

Gaunt A, Patel A, Rusius V, Royle TJ, Markham DH, Pawlikowska T. “Playing the game”: how do surgical trainees seek feedback using workplace-based assessment? Med Educ. 2017;51(9):953–62.

Martin L, Sibbald M, Brandt Vegas D, Russell D, Govaerts M. The impact of entrustment assessments on feedback and learning: trainee perspectives. Med Educ. 2020;54(4):328–36.

Bray MJ, Bradley EB, Martindale JR, Gusic ME. Implementing systematic faculty development to support an EPA-Based program of assessment: strategies, outcomes, and lessons learned. Teach Learn Med. 2021;33(4):434–44.

Lövquist E, Shorten G, Aboulafia A. Virtual reality-based medical training and assessment: the multidisciplinary relationship between clinicians, educators and developers. Med Teach. 2012;34(1):59–64.

Norcini JJ. Peer assessment of competence. Med Educ. 2003;37(6):539–43.

Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE guide no. 31. Med Teach. 2007;29(9):855–71.

Slattery P, Saeri AK, Bragge P. Research co-design in health: a rapid overview of reviews. Health Res Policy Syst. 2020;18(1):17.

Download references

Acknowledgements

The authors would like to acknowledge the contribution of Mr. Guy Gielis, Mrs. An Stockmans, Mrs. Fran Timmers, and Mrs Karolina Bystram that assisted with coordination of the CCCs. We would also like to thank and acknowledge Prof. dr. Martin Valcke and Dr. Mieke Embo for facilitating this study through the SBO SCAFFOLD project(www.sbo-scaffold.com). Finally, we would like to thank the CCCs members and the trainers and trainees that participated in this study.

This work was supported by the Research Foundation Flanders (FWO) under Grant [S003219N]-SBO SCAFFOLD.

Author information

Authors and affiliations.

Academic Centre for General Practice, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium

Vasiliki Andreou, Sanne Peters & Birgitte Schoenmakers

School of Health Sciences, Faculty of Medicine, Dentistry and Health Sciences, The University of Melbourne, Melbourne, Australia

Sanne Peters

Department of Cellular and Molecular Medicine, KU Leuven, Leuven, Belgium

Jan Eggermont

Department of Public Health and Primary Care, KU Leuven, Box 7001, Kapucijnenvoer 7, Leuven, 3000, Belgium

Vasiliki Andreou

You can also search for this author in PubMed   Google Scholar

Contributions

All authors (VA, SP, JE, BS) have contributed to designing the study. VA collected the data, led the analysis, and wrote the manuscript. BS analysed the data and critically reviewed the manuscript. SE and JE contributed to critically revising this manuscript. All authors have read and approved the manuscript.

Corresponding author

Correspondence to Vasiliki Andreou .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by the Social and Societal Ethics Committee G-2022-5615-R2(MIN) , and all participants signed a informed consent prior to participation.

Consent for publication

All the participants gave their consent for publishing their data anonymously.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Andreou, V., Peters, S., Eggermont, J. et al. Co-designing Entrustable Professional Activities in General Practitioner’s training: a participatory research study. BMC Med Educ 24 , 549 (2024). https://doi.org/10.1186/s12909-024-05530-y

Download citation

Received : 25 December 2023

Accepted : 07 May 2024

Published : 17 May 2024

DOI : https://doi.org/10.1186/s12909-024-05530-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Postgraduate medical education
• Curriculum design
• EPA assessment
• GP Training
• Workplace-based assessment

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 16.5.2024 in Vol 11 (2024)

Psychometric Assessment of an Item Bank for Adaptive Testing on Patient-Reported Experience of Care Environment for Severe Mental Illness: Validation Study

Authors of this article:

Original Paper

• Sara Fernandes 1 , PhD   ; 
• Yann Brousse 1 , MSc   ; 
• Xavier Zendjidjian 1 , MD, PhD   ; 
• Delphine Cano 1 , MSc   ; 
• Jérémie Riedberger 1 , MSc   ; 
• Pierre-Michel Llorca 2 , MD, PhD   ; 
• Ludovic Samalin 2 , MD, PhD   ; 
• Daniel Dassa 1 , MD, PhD   ; 
• Christian Trichard 3 , MD, PhD   ; 
• Vincent Laprevote 4, 5 , MD, PhD   ; 
• Anne Sauvaget 6 , MD, PhD   ; 
• Mocrane Abbar 7 , MD, PhD   ; 
• David Misdrahi 8 , MD, PhD   ; 
• Fabrice Berna 5 , MD, PhD   ; 
• Christophe Lancon 1 , MD, PhD   ; 
• Nathalie Coulon 9 , MD, PhD   ; 
• Wissam El-Hage 10 , MD, PhD   ; 
• Pierre-Emmanuel Rozier 11 , MD   ; 
• Michel Benoit 12 , MD, PhD   ; 
• Bruno Giordana 12 , MD, PhD   ; 
• Alejandra Caqueo-Urízar 13 , PhD   ; 
• Dong Keon Yon 14 , MD, PhD   ; 
• Bach Tran 1, 15 , PhD   ; 
• Pascal Auquier 1 , MD, PhD   ; 
• Guillaume Fond 1 , MD, PhD   ; 
• Laurent Boyer 1 , MD, PhD  

1 Assistance Publique-Hopitaux de Marseille, Aix-Marseille University, UR3279: Health Service Research and Quality of Life Center - CEReSS, Marseille, France

2 Department of Psychiatry, Centre Hospitalier Universitaire de Clermont-Ferrand, University of Clermont Auvergne, Centre national de la recherche scientifique, Institut national polytechnique de Clermont Auvergne, Institut Pascal UMR 6602, Clermont-Ferrand, France

3 Etablissement public de santé Barthélemy Durand, Etampes, France

4 Department of Addictology and Psychiatry, Centre Psychothérapique de Nancy, Laxou, France

5 Département de Psychiatrie, Centre Hospitalier Régional Universitaire de Strasbourg, Université de Strasbourg, Institut national de la santé et de la recherche médicale U1114, Fédération de Médecine Translationnelle de Strasbourg, Strasbourg, France

6 Nantes Université, Centre Hospitalier Régional Universitaire de Nantes, Movement - Interactions - Performance - MIP UR 4334, Nantes, France

7 Department of Psychiatry, Centre Hospitalier Universitaire de Nîmes, University of Montpellier, Nîmes, France

8 National Centre for Scientific Research UMR 5287 - Institut de Neurosciences Cognitives et Intégratives d'Aquitaine, University of Bordeaux, Centre Hospitalier Charles Perrens, Bordeaux, France

9 Centre Expert Schizophrénie, Centre Expert TSA-SDI et Centre Référent de Réhabilitation Psychosociale et de Remédiation Cognitive - C3R, Centre Hospitalier Alpes Isère, Grenoble, France

10 Centre Hospitalier Régional Universitaire de Tours, Clinique Psychiatrique Universitaire, Tours, France

11 Department of Psychiatry, Centre Hospitalier des Pyrénées, Pau, France

12 Department of Psychiatry, Hopital Pasteur, University Hospital of Nice, Nice, France

13 Instituto de Alta Investigación, Universidad de Tarapacá, Arica, Chile

14 Center for Digital Health, Medical Science Research Institute, Kyung Hee University College of Medicine, Department of Pediatrics, Kyung Hee University Medical Center, Seoul, Republic of Korea

15 Institute of Preventive Medicine and Public Health, Hanoi Medical University, Hanoi, Vietnam

Corresponding Author:

Sara Fernandes, PhD

Assistance Publique-Hopitaux de Marseille, Aix-Marseille University, UR3279: Health Service Research and Quality of Life Center - CEReSS

27, Boulevard Jean-Moulin

Marseille, 13385

Phone: 33 660185077

Email: [email protected]

Background: The care environment significantly influences the experiences of patients with severe mental illness and the quality of their care. While a welcoming and stimulating environment enhances patient satisfaction and health outcomes, psychiatric facilities often prioritize staff workflow over patient needs. Addressing these challenges is crucial to improving patient experiences and outcomes in mental health care.

Objective: This study is part of the Patient-Reported Experience Measure for Improving Quality of Care in Mental Health (PREMIUM) project and aims to establish an item bank (PREMIUM-CE) and to develop computerized adaptive tests (CATs) to measure the experience of the care environment of adult patients with schizophrenia, bipolar disorder, or major depressive disorder.

Methods: We performed psychometric analyses including assessments of item response theory (IRT) model assumptions, IRT model fit, differential item functioning (DIF), item bank validity, and CAT simulations.

Results: In this multicenter cross-sectional study, 498 patients were recruited from outpatient and inpatient settings. The final PREMIUM-CE 13-item bank was sufficiently unidimensional (root mean square error of approximation=0.082, 95% CI 0.067-0.097; comparative fit index=0.974; Tucker-Lewis index=0.968) and showed an adequate fit to the IRT model (infit mean square statistic ranging between 0.7 and 1.0). DIF analysis revealed no item biases according to gender, health care settings, diagnosis, or mode of study participation. PREMIUM-CE scores correlated strongly with satisfaction measures ( r =0.69-0.78; P <.001) and weakly with quality-of-life measures ( r =0.11-0.21; P <.001). CAT simulations showed a strong correlation ( r =0.98) between CAT scores and those of the full item bank, and around 79.5% (396/498) of the participants obtained a reliable score with the administration of an average of 7 items.

Conclusions: The PREMIUM-CE item bank and its CAT version have shown excellent psychometric properties, making them reliable measures for evaluating the patient experience of the care environment among adults with severe mental illness in both outpatient and inpatient settings. These measures are a valuable addition to the existing landscape of patient experience assessment, capturing what truly matters to patients and enhancing the understanding of their care experiences.

Trial Registration: ClinicalTrials.gov NCT02491866; https://clinicaltrials.gov/study/NCT02491866

Introduction

The health care environment, which encompasses design features (ie, cleanliness, food, privacy, waiting time, basic amenities) and the overall atmosphere (or climate) [ 1 ], has been recognized as a significant factor influencing the experiences of patients with severe mental illness (SMI) [ 2 - 5 ]. It is an important factor in the quality of patient care [ 2 , 6 - 8 ], contributing to improved patient satisfaction [ 9 ] and improved health outcomes [ 10 , 11 ]. In a recent study, patients identified a welcoming environment as one of the most important aspects of their care [ 12 ]. Indeed, a calm and welcoming environment helps to improve patients’ sense of control and empowerment and, consequently, reinforces their willingness to follow recommended treatments. In addition, the care environment is the patients’ first impression and can lead to a positive image of the therapeutic process [ 13 ]. A supportive environment promotes communication between patients and staff, can help reduce stressful stimuli, and thus prevents relapses and risky behavior. The priority for psychiatric facilities is therefore to provide patients with a warm and safe atmosphere that allows for positive social interactions, with opportunities for stimulating activities, enabling patients to facilitate their recovery and transition to the community. Different theoretical models can shed light on the additional nonpharmacological and biopsychosocial effects of a patient’s care experience, including the placebo response effects and the set and setting theory [ 14 , 15 ].

Recommended features to promote patient recovery [ 16 - 19 ] include smaller, home-like units with well-decorated common spaces, open designs, access to nature and daylight, and an environment that is clean, well laid out, and ensures privacy and security for personal effects. However, psychiatric facilities are often criticized for prioritizing staff workflow over patient needs [ 2 ], leading in some cases to a perceived “prison-like atmosphere” [ 16 , 20 , 21 ] characterized by conflicting routines and rules and a lack of stimulation [ 22 , 23 ]. Some patients have reported feelings of boredom, loneliness, and stigmatization in these environments [ 21 - 26 ]. The lack of stimulating activities and positive social interactions is a barrier to patients’ successful recovery [ 24 - 30 ]. These negative experiences can contribute to decreased patient satisfaction, increased levels of anxiety and stress among patients, ineffective care, and signs of burnout among staff [ 27 , 31 , 32 ]. Emphasis should be placed on the design of psychiatric facilities, as a difficult environment is a barrier to care, and patients often perceive such an environment as a lack of attention from staff [ 30 ]. In psychiatry, patients cope with an unfamiliar and potentially stressful environment [ 33 ], and a better understanding of their experiences is essential to identify and improve current barriers.

Given this growing interest, it is necessary to provide a valid and reliable instrument for measuring patients’ experience of the care environment, applicable to both inpatient and outpatient settings, as care pathways for patients with SMI often combine several care modalities. Previous research has demonstrated that patients with SMI can provide reliable and valid responses to self-administered questionnaires; the impact of psychiatric symptoms and cognitive deficits seems to be negligible [ 34 , 35 ]. The French group PREMIUM (Patient-Reported Experience Measure for Improving Quality of Care in Mental Health) is developing item banks and computerized adaptive tests (CATs) to improve the systematic use of patient-reported experience measures in mental health care [ 36 ]. The use of CATs significantly reduces measurement burden by administering a limited number of items targeted to the respondent’s experience level, aiming to improve measurement accuracy.

The objective of this study was to calibrate an item bank and develop a CAT to assess the care environment experienced by adult patients with SMI. These measures will contribute to the current landscape of patient experience measures by providing a valuable complement to PREMIUM measures and capturing what really matters to patients.

Study Population and Procedure

This is a national, multicenter, cross-sectional study conducted between January 2016 and December 2021. Patients were recruited through in- and outpatient psychiatric settings of a French teaching hospital (Assistance Publique-Hôpitaux de Marseille), the FondaMental Foundation’s expert centers [ 37 ], and through an online survey. In mental health settings, stable patients who met the inclusion criteria were identified and approached by a member of their usual care team to invite them to participate in the study. The link to the web survey was distributed through patient associations.

Inclusion criteria were as follows: age older than 18 years and younger than 65 years with a diagnosis of schizophrenia, bipolar disorder, or major depressive disorder (MDD), receiving inpatient or outpatient psychiatric care, and speaking or reading French. Vulnerable persons (ie, pregnant or nursing women, persons under legal protection) or those unable to complete a self-administered questionnaire were not included in the study.

Current recommendations suggest a sample size of 300-500 observations for multiparameter item response theory (IRT) models [ 38 - 40 ]. Consequently, we estimated that a sample of around 500 patients would be sufficient to obtain reasonably stable estimates.

Data Collection

Data were collected through paper questionnaires in health care settings and online through a web survey. Patients reported the following sociodemographic and clinical characteristics: gender, age, educational level, marital status, occupational status, main diagnosis (schizophrenia, bipolar disorders, or MDD), duration of illness, and quality of life (QoL) as measured using the medical outcome study 12-item Short Form (SF-12) [ 41 ], which describes 8 QoL dimensions: physical functioning, social functioning, role physical, role emotional, mental health, vitality, bodily pain, general health, and 2 composite scores for physical and mental QoL (ranging from 0 to 100, with higher scores indicating better QoL). Adequate psychometric properties of the SF-12 have been demonstrated among individuals with SMI [ 42 ], and the SF-12 has proven to be a good alternative to the SF-36 for minimizing response burden.

The PREMIUM for Care Environment (PREMIUM-CE) item bank consists of 16 items designed for patients with SMI and measures their experience regarding the care environment over the past 4 weeks. Participants respond to the items on a 5-point Likert scale ranging from “strongly disagree” to “strongly agree” with a “not applicable” response option. Additionally, an overall satisfaction item (“Overall, are you satisfied with the health care facilities in which you receive care?”) and a visual analog scale (VAS; minimum 0 to maximum 10) were collected. PREMIUM-CE items were identified through face-to-face interviews with patients with SMI and a systematic review of existing patient-reported experience measure; then the item pool was refined based on an expert review and cognitive interviews with patients with SMI [ 4 , 5 , 36 ].

Statistical Analysis

Basic descriptive analysis.

Descriptive statistics were calculated to describe participants’ characteristics, including frequencies and percentages for categorical variables and means and SDs for continuous variables. Response rates, means and SDs, and ceiling and floor effects were also calculated for each item.

IRT Assumptions

Unidimensionality, local independence, and monotonicity are the 3 fundamental assumptions of IRT [ 43 ]. Data were randomly divided into 2 data sets (n=249 each), one for exploratory factor analysis and one for confirmatory factor analysis (CFA) with the weighted least squares mean and variance estimator to ensure that the PREMIUM-CE was sufficiently unidimensional [ 44 ]. Local independence was examined using residual correlations from the final CFA model. Monotonicity was examined using visual inspection of characteristic item curves.

Calibration and Fitting an IRT Model

Item parameters were estimated using the generalized partial credit model (GPCM) [ 45 ] and compared to the partial credit model [ 46 ]. IRT handles missing values by using full information maximum likelihood estimation, which uses all available information, and GPCM is recommended when the amount of missing data is high (20% or more) [ 38 ]. Item fit was assessed by examining the mean square infit statistics, which reflect the information-weighted mean squared residuals between the observed and expected response patterns. PREMIUM-CE scores (θ) were estimated by the Bayesian Expected a Posteriori estimation method [ 47 ], and a linear transformation was performed to obtain PREMIUM-CE scores ranging from 0 to 100 (higher scores indicate better experience with the care environment). The information curve of the final item bank was calculated, and high measurement precision was defined as an information score >10, corresponding to a reliability of >0.90 [ 39 ].

Differential Item Functioning Analysis

DIF was examined using ordinal logistic regression models [ 48 , 49 ] by gender (man vs woman), age (median distribution), care setting (outpatient vs inpatient), psychiatric diagnosis (schizophrenia vs bipolar disorder vs MDD), and mode of study participation (online survey vs health care settings).

External Validity

Construct validity was examined through convergent and discriminative validity assessments. For convergent validity, Spearman’s rank correlations were computed between PREMIUM-CE scores and both satisfaction (global satisfaction item and VAS) and QoL (SF-12 subscales and composite scores) scores. Our hypothesis was that PREMIUM-CE scores would have strong correlations with satisfaction scores ( r >0.60), which are 2 related measures, and weak correlations with QoL scores ( r <0.30). For discriminant validity, relationships between PREMIUM-CE scores and sociodemographic and clinical characteristics of the respondents were examined by using 2-tailed t tests, ANOVA, and Pearson correlations. The Q-Q plot was used to determine that the data are approximately normally distributed. Based on previous studies of the determinants of patient satisfaction with psychiatric services [ 3 , 50 , 51 ], our hypotheses were that higher levels of patient experience of the care environment were associated with older age, being female, being nonsingle, or being in an outpatient setting.

CAT Simulations

These simulations using participants’ actual responses were run from the calibrated item bank and compared to identify the best performing CAT version. The stopping rules were based on standard error of measurement (SEM) values of 0.33, 0.44, and 0.55 (corresponding to a reliability between 0.90 and 0.70 [ 52 ]). The item administered at baseline was the one that offered the most information to the population mean (θ=0), and then items were administered according to the maximum Fisher information criterion [ 53 ].

The indicators used at each stage of the psychometric analyses are presented in Table S1 in Multimedia Appendix 1 [ 44 , 54 - 70 ]. All of the statistical analyses were performed using the following software: SPSS (version 20.0; IBM Corp), MPlus (version 7.0; Muthen & Muthen), and R (version 4.2.0; R Core Team), using packages mirt [ 71 ], lordif [ 72 ], BifactorIndicesCalculator [ 73 ], and mirtCAT [ 74 ]. A 2-tailed P <.05 was considered statistically significant.

Ethical Considerations

This study was conducted in accordance with the Declaration of Helsinki and approved by the relevant ethics committee (2014-A01152-45). The study was registered in ClinicalTrials.gov (NCT02491866). All participants provided nonopposition, as required by French law. Additionally, all data were anonymized.

Sample Characteristics

Of the 498 participants, 50.2% (250/498) were men, 72.3% (345/477) were unemployed, 73.5% (350/476) were single, and 70.5% (337/478) had an education level of a bachelor’s degree or higher. The average age was 40.9 (SD 11.9) years, and the mean duration of illness was 12.9 (SD 9.3) years. In total, 51.8% (253/488) of the participants had a diagnosis of schizophrenia, 24.4% (119/488) had bipolar disorder, or 23.8% (116/488) had MDD, and 77.7% (387/497) of them were outpatients. The characteristics of the sample are presented in Table 1 .

a SF-12: 12-item Short Form.

Basic Descriptive Statistics

The mean item scores ranged from 2.07 (SD 1.32) to 3.24 (SD 0.89), and most items had a missing data rate <10% (except items CE10, CE12, and CE15). The floor and ceiling effects ranged from 1.8% to 10.6% and from 10% to 45.2%, respectively. The interitem correlation values ranged from 0.01 to 0.79, and 3 pairs of items showed too high interitem correlations (>0.70): items CE3-CE4 ( r =0.73), items CE3-CE5 ( r =0.78), and items CE4-CE5 ( r =0.79). Items CE3 and CE5 were excluded because their content was considered less relevant than the remaining items. The lowest scores were for item CE15 (“food was of good quality”), item CE12 (“you had access to media (telephone, computer, internet or Wi-Fi connection, etc),” and item CE10 (“the health care facilities were well equipped”). Table 2 summarizes the distribution of responses for each item.

In EFA, 2 factors had eigenvalue greater than 1, and the scree plot and parallel analysis indicated 2 factors. The eigenvalue of the first factor was 6.46 and explained 46.11% of the total variance; the second factor was 1.33, and the ratio was 4.86. Evaluations indicated that the 2 spatial accessibility items (CE1 and CE2) may form a separate factor, and after a content review, only item CE1 was kept as it was deemed the most relevant. The 1-factor CFA model provided evidence to support the unidimensionality of the remaining 13 items (root mean square error of approximation [RMSEA]=0.082; 95% CI 0.067-0.097; comparative fit index=0.974; Tucker-Lewis index=0.968) and no items showed local dependence (all residual correlations were above |0.20|). Of the 13 items in the bank, 10 were recoded to meet the monotonicity assumption (Table S2 in Multimedia Appendix 1 ), which improved the model fit (Akaike information criterion=–3343.78 and Bayes information criterion=–3428). Cronbach α was .91.

The GPCM was used to calibrate the item bank and showed superior fit compared to the partial credit model (10,192.60 and 10,367.43 for Akaike information criterion and 10,382.07 and 10,506.38 for Bayes information criterion; and χ 2 =198.84; P <.001); item fit was good (infit values ranging between 0.74 and 1.00). IRT parameter estimates for the 13 items showed slopes ranging from 0.55 to 2.85 and thresholds ranging from –2.07 to 2.29. Item parameters and item fit are provided in Table S2 in Multimedia Appendix 1 . As shown in Figure 1 , PREMIUM-CE provided the most information in the scale range between –2.6 and 1.4 and had a high measurement accuracy (reliability >0.90) in a shorter range between –2.1 and 0.7 (which corresponds to 88.6% of total information). Item CE7 was the most informative of the bank—“the health care facilities were comfortable,” whereas item CE15 was the least informative—“the food was of good quality.”

Responses to items CE6 (quiet) and CE13 (sanitary) were flagged for overall DIF but with negligible magnitude according to health care settings. Likewise, the DIF magnitude was negligible for item CE16 (smoking ban) according to mode of study participation and for item CE15 (food) according to gender, mode of study participation, and diagnostic after pooling bipolar disorder and MDD (mood disorders vs schizophrenia; P =.02; Δ R 2 =.013). None of the items showed significant DIF for age. DIF results are provided in Table S3 in Multimedia Appendix 1 .

As expected, there were strong correlations between the PREMIUM-CE item bank and overall satisfaction and VAS, supporting convergent validity. Similarly, all SF-12 dimensions were weakly correlated with the PREMIUM-CE item bank, except for bodily pain and vitality. Associations were found between better experience of the care environment (ie, higher PREMIUM-CE scores) and older age, being a woman, being voluntarily admitted to a hospital, and being recruited through health care facilities. There was no significant effect of educational level, marital status, employment status, diagnosis, or duration of illness. These results are presented in Table 3 .

a N/A: not applicable.

b SF-12: 12- items short form.

As reported in Table 4 , the results of the CAT simulations based on SEM <.33 and <.44 were both acceptable in terms of accuracy and precision, but the scenario based on SEM <.33 (corresponding to a reliability of 0.90) was the most efficient. Of the 498 participants included in the simulation, 79.5% (396) achieved a reliable score with an average of 7 items administered.

a SEM: standard error of measurement.

b RMSE: root mean square error.

Principal Findings

In this study, we report the calibration and initial evaluation of a new PREMIUM-CE item bank measuring patients’ experience of the care environment that can be used for CATs. The PREMIUM-CE questionnaire is the first available questionnaire thus far to assess the quality of the care environment, applicable in outpatient and inpatient settings, for adults with SMI. This new measure covers different facets of the care environment, including ease of access in time and space, facility layout and basic amenities, food quality, comfort and cleanliness, respect for privacy, and smoking ban. PREMIUM-CE items address both concerns common to all patients (eg, cleanliness or food) and those more specific to psychiatric patients (eg, therapeutic workshops). Existing instruments measure more objective aspects (eg, checklists fulfilled by direct observation), and patients with SMI were not involved in the development and validation process [ 75 ].

PREMIUM-CE has undergone rigorous psychometric evaluation, consistent with previous studies conducted as part of the French PREMIUM initiative [ 36 ]. Although the RMSEA was slightly above the criterion of <.08, our results provide evidence of sufficient unidimensionality, and the item pool meets the assumptions for IRT modeling. Research has shown that the RMSEA statistic is problematic for assessing the unidimensionality of item banks measuring health concepts [ 76 ], as RMSEA is sensitive to model complexity (number of estimated parameters) and skewed data distributions [ 77 ]. These results are comparable to other calibration studies of item banks of patient-reported measures [ 78 - 83 ]. Overall, our results demonstrate that PREMIUM-CE has strong psychometric properties for patients with SMI, with negligible measurement bias by gender, health care settings, and mode of study participation. Items CE10, CE12, and CE15 had a higher rate of missing data than the other items, but this rate was below 40%, which remains acceptable by psychometrics standards [ 84 ]. In addition, these items had lower scores compared to others, meaning that efforts should be targeted on these aspects to improve the experience of patients with SMI. Future studies should examine whether changes to these items are required. The absence of a large DIF magnitude according to health care settings will make it possible to study changes in the experience of psychiatric patients over time, for whom care pathways often combine inpatient and outpatient care modalities. The 13 items in the final version of the PREMIUM-CE are listed in Multimedia Appendix 1 , Table S4. In addition, the CAT version showed comparable measurement accuracy to the full item bank with high correlations between scores with an average of only 7 items administered.

External validity, explored using validated questionnaires and sociodemographic and clinical data, generally supported our initial hypotheses. Previous research has demonstrated that some factors, such as age, gender, marital status, and physical and mental health status, can influence individuals’ experiences within a specific environment [ 2 , 3 ]. It is important to note, however, that the literature has not consistently established clear associations for age, gender, and marital status [ 3 ]. According to our results, older age, being female, being voluntarily admitted, and reporting a good physical and mental quality of life are associated with higher levels of patient experience of the care environment. As previously described [ 85 ], women reported higher levels of experience than men. Likewise, older people tend to be more accommodating, perhaps because they have fewer expectations than younger people [ 86 ]. Also, contrary to what might be expected, voluntarily admitted patients reported higher levels of experience than outpatients, although some patients reported a preference for community mental health treatment, which they considered less stigmatizing [ 87 ] and compatible with professional and social functioning. Furthermore, the literature has shown that hospitalization, particularly in the context of involuntary admission, can have a negative impact on patient experience [ 3 ], because it can be experienced as traumatic or particularly stressful for patients [ 88 ].

However, our results suggest that patients voluntarily admitted to the hospital may have a more holistic and structured experience compared to outpatients, conducive to positive therapeutic relationships with staff, whereas constraint has a negative effect on therapeutic relationships in the case of involuntarily admitted patients [ 88 - 90 ]. Finally, a positive but weak association was found between higher levels of patient experience and better QoL, as previously reported in other studies [ 51 ]. A calm and welcoming care environment contributes to patients feeling more comfortable and safer [ 50 ], which can reduce stress and anxiety and enhance relationships with staff, thereby promoting patients’ recovery. Participants completing the online survey reported a poorer experience of the care environment than participants in health care settings because the latter may be more favorable due to fear of a negative effect on their relationships with staff, or this difference may be due to a possible recall bias.

The most poorly rated items by patients were related to accessing equipment (CE10), media (CE12), and food (CE15). Difficulties with access to equipment (eg, for art therapy) and media (eg, televisions or computers) are related to boredom, isolation, frustration, and higher levels of distress in patients [ 25 ]. A variety of individual or group activities could be offered to patients, such as therapeutic workshops in self-expression (ie, writing), art (ie, photography or painting), psychosocial rehabilitation (ie, cooking, which may also improve diet habits), or body awareness (ie, sophrology), to help patients develop social skills and promote social reintegration, improve confidence and self-esteem, build emotional resilience, and enjoy themselves. Facilities should have basic amenities such as affordable Wi-Fi and a working television in a common room accessible to all patients. Likewise, rooms should be equipped with a minimum package of free channels; however, not all facilities are equally equipped, and the cost of access to Wi-Fi and pay television channels can vary by as much as 2-fold. The content of what is broadcast on television should also be a therapeutic consideration. For example, it seems logical to avoid broadcasting distressing news or uninspiring programs and to favor the broadcasting of cultural works that could be the object of an exchange after viewing, such as a film club. The use of cell phones in health care settings presents challenges in terms of the potential risk of theft or breakage, as well as concerns about maintaining confidentiality. Additionally, it can be a source of tension with staff (eg, if the telephone credit is exceeded). There is no law that prohibiting the use of cell phones because communicating is a fundamental individual freedom, but the internal rules of the facilities can regulate their use by specifying the times and places of use and prohibit taking pictures of patients and staff. Furthermore, psychiatrists may occasionally prohibit a patient from keeping a cell phone, computer, or tablet as part of a medical decision, particularly in the case of placement in a seclusion room or for medical conditions. Previous studies have shown that a healthy diet is essential for good mental health and can prevent the worsening of symptoms [ 91 , 92 ], and that patients’ satisfaction with hospital food services strongly influences their overall satisfaction with hospital care [ 93 ]. Diets such as the Mediterranean diet have been shown to improve patient outcomes [ 91 ]. Providing a menu tailored to patient preferences while focusing on food quality (taste, presentation, flavor, preparation, and variety), as well as the hospital environment, will help improve inpatient appetite and satisfaction [ 93 ]. In summary, the current challenges of hospital food service are to transition to a diet that is lower in meat, closer to the Mediterranean diet, without plastic packaging, and low in processed products while increasing the attractiveness of local and seasonal products, all while maintaining costs [ 91 ]. By contrast, the most highly rated items by patients were related to spatial accessibility (CE1), cleanliness (CE8), and smoking ban (CE16). Although health care facilities are under a total smoking ban throughout their whole facilities (including in specifically dedicated “smoking areas” or outside), the reality is often more flexible to accommodate patients who cannot leave the health care facilities, even temporarily (eg, patients under constraint). Proposals for smoking cessation assistance should be systematically offered to patients.

Limitations

Some limitations of this study are worth noting. Our sample size, while relatively modest, was sufficient to obtain accurate estimates. Current recommendations suggest that at least 300 observations are sufficient when using multiparameter models like the GPCM [ 38 - 40 ]. However, our results showed that the assumptions required for IRT calibration were met and that the model fit was adequate. In addition, some DIF analyses comparing subgroups with sample sizes smaller than those recommended for DIF analyses (at least 200 observations per group [ 94 ]) may have lacked the statistical power to detect a statistically significant DIF. These DIF findings should be regarded as preliminary, and future work with a larger sample will allow us to confirm these results. Although participants from the online survey and those from health care settings may have reported different levels of experience, this mixed survey design was chosen to ensure inclusivity across various subgroups, as supported by previous research on the equivalence of administration methods [ 95 ]. DIF analysis revealed that none of the items was flagged with a large DIF magnitude according to the patient’s mode of study participation, suggesting that the data can be pooled without substantial bias. It was not possible to calculate a participation rate or to compare the characteristics of respondents and nonrespondents. This study was widely disseminated nationally, and our sample included inpatients and outpatients with diverse characteristics from different geographic regions of the country. Patients self-reported their diagnosis, and some data (488/498, 2.5%) were missing. However, the risk of misdiagnosis is considered minimal because all participants were fully informed about the study scope and diagnostic criteria. Additionally, this approach closely mirrors the real-world conditions of PREMIUM use. The title of the study mentioned general experience of care to limit the self-selection bias of patients with extreme care environment experiences. Future work will confirm the generalizability of our results. PREMIUM-CE has greater measurement accuracy for patients with scores between –2.1 and +0.7 (ie, reporting low to moderate levels of experience), and thus more items are needed to estimate scores for patients at both ends of the latent continuum. Future work should also reevaluate the precision and accuracy of the CAT in an independent sample and under real-world conditions. Finally, criterion validity could not be assessed because, to our knowledge, no gold standard was available and evidence for construct validity was limited. Future validation studies should examine the relationship between this new measure and objective assessments of the care environment (eg, evaluation by architects or other professionals).

The PREMIUM-CE item bank and its CAT version have demonstrated strong psychometric properties, making them robust measures for assessing patient experience of the care environment, applicable in both outpatient and inpatient settings, for adults with SMI. These measures contribute to the current landscape of patient experience measures by providing a valuable complement to PREMIUM measures of what really matters to patients.

Acknowledgments

The authors wish to thank Baumstarck Karine, Boucekine Mohamed, Loundou Anderson, and Michel Pierre for their statistical and methodological support. This research received funding through institutional grants from the French Programme de recherche sur la performance du système des soins and the Agence Technique de l’Information sur l’Hospitalisation. The sponsors had no role in study design, collection, analysis, and interpretation of data, report writing, or the decision to submit the study for publication.

Authors' Contributions

LB was responsible for the conceptualization, supervision, project administration, and funding acquisition. SF was responsible for the methodology, formal analysis, and writing the original draft. LB, SF, and GF are responsible for review and editing. All the authors read and agreed to the published version of the manuscript.

Conflicts of Interest

None declared.

Indicators of psychometric performance, parameter estimates (discrimination and thresholds) and fit statistics, differential item functioning results, and list of the 13-item of the Patient-Reported Experience Measure for Improving Quality of Care in Mental Health for Care Environment (PREMIUM-CE) item bank (English and French versions).

• Nicholls D, Kidd K, Threader J, Hungerford C. The value of purpose built mental health facilities: use of the Ward Atmosphere Scale to gauge the link between milieu and physical environment. Int J Ment Health Nurs. 2015;24(4):286-294. [ CrossRef ] [ Medline ]
• Daigle K, Frankel L. Collaboration to improve experience in hospital environments. In: Di Bucchianico G, Shin CS, Shim S, Fukuda S, Montagna G, Carvalho C, editors. Advances in Industrial Design. Cham, Switzerland. Springer; 2020:403-409.
• Woodward S, Berry K, Bucci S. A systematic review of factors associated with service user satisfaction with psychiatric inpatient services. J Psychiatr Res. 2017;92:81-93. [ CrossRef ] [ Medline ]
• Fernandes S, Fond G, Zendjidjian XY, Baumstarck K, Lançon C, Berna F, et al. Measuring the patient experience of mental health care: a systematic and critical review of patient-reported experience measures. Patient Prefer Adherence. 2020;14:2147-2161. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fernandes S, Fond G, Zendjidjian X, Michel P, Lançon C, Berna F, et al. A conceptual framework to develop a patient-reported experience measure of the quality of mental health care: a qualitative study of the PREMIUM project in France. J Mark Access Health Policy. 2021;9(1):1885789. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Henriksen K, Isaacson S, Sadler BL, Zimring CM. The role of the physical environment in crossing the quality chasm. Jt Comm J Qual Patient Saf. 2007;33(Suppl 11):68-80. [ CrossRef ] [ Medline ]
• Zhao Y, Mourshed M. Patients’ perspectives on the design of hospital outpatient areas. Buildings. 2017;7(4):117. [ FREE Full text ] [ CrossRef ]
• Connellan K, Gaardboe M, Riggs D, Due C, Reinschmidt A, Mustillo L. Stressed spaces: mental health and architecture. HERD. 2013;6(4):127-168. [ CrossRef ] [ Medline ]
• Jovanović N, Miglietta E, Podlesek A, Malekzadeh A, Lasalvia A, Campbell J, et al. Impact of the hospital built environment on treatment satisfaction of psychiatric in-patients. Psychol Med. 2022;52(10):1969-1980. [ CrossRef ] [ Medline ]
• Ulrich RS, Zimring C, Zhu X, DuBose J, Seo HB, Choi YS, et al. A review of the research literature on evidence-based healthcare design. HERD. 2008;1(3):61-125. [ CrossRef ] [ Medline ]
• Jamshidi S, Parker JS, Hashemi S. The effects of environmental factors on the patient outcomes in hospital environments: a review of literature. Front Archit Res. 2020;9(2):249-263. [ FREE Full text ] [ CrossRef ]
• Kelly EL, Davis L, Mendon S, Kiger H, Murch L, Pancake L, et al. Provider and consumer perspectives of community mental health services: implications for consumer-driven care. Psychol Serv. 2019;16(4):572-584. [ CrossRef ] [ Medline ]
• Ghaffari F, Shabak M, Norouzi N, Fallah SN. Hospital salutogenic public spaces: a conceptual framework of effective perceptional environment quality components on patients' satisfaction. Int J Build Pathol Adapt. 2021;41(5):965-987. [ CrossRef ]
• Colloca L, Barsky AJ. Placebo and nocebo effects. N Engl J Med. 2020;382(6):554-561. [ CrossRef ] [ Medline ]
• Hartogsohn I. Set and setting, psychedelics and the placebo response: an extra-pharmacological perspective on psychopharmacology. J Psychopharmacol. 2016;30(12):1259-1267. [ CrossRef ] [ Medline ]
• Jovanović N, Campbell J, Priebe S. How to design psychiatric facilities to foster positive social interaction—a systematic review. Eur Psychiatry. 2019;60:49-62. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ulrich RS, Bogren L, Gardiner SK, Lundin S. Psychiatric ward design can reduce aggressive behavior. J Environ Psychol. 2018;57:53-66. [ CrossRef ]
• Shepley MM, Watson A, Pitts F, Garrity A, Spelman E, Kelkar J, et al. Mental and behavioral health environments: critical considerations for facility design. Gen Hosp Psychiatry. 2016;42:15-21. [ CrossRef ] [ Medline ]
• Borge L, Fagermoen MS. Patients' core experiences of hospital treatment: wholeness and self-worth in time and space. J Ment Health. 2009;17(2):193-205. [ CrossRef ]
• Staniszewska S, Mockford C, Chadburn G, Fenton SJ, Bhui K, Larkin M, et al. Experiences of in-patient mental health services: systematic review. Br J Psychiatry. 2019;214(6):329-338. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shattell MM, Andes M, Thomas SP. How patients and nurses experience the acute care psychiatric environment. Nurs Inq. 2008;15(3):242-250. [ CrossRef ] [ Medline ]
• Molin J, Graneheim UH, Lindgren BM. Quality of interactions influences everyday life in psychiatric inpatient care--patients' perspectives. Int J Qual Stud Health Well-being. 2016;11:29897. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lindgren BM, Aminoff C, Graneheim UH. Features of everyday life in psychiatric inpatient care for self-harming: an observational study of six women. Issues Ment Health Nurs. 2015;36(2):82-88. [ CrossRef ] [ Medline ]
• Lilja L, Hellzén O. Former patients' experience of psychiatric care: a qualitative investigation. Int J Ment Health Nurs. 2008;17(4):279-286. [ CrossRef ] [ Medline ]
• Folke F, Hursti T, Kanter JW, Arinell H, Tungström S, Söderberg P, et al. Exploring the relationship between activities and emotional experience using a diary in a mental health inpatient setting. Int J Ment Health Nurs. 2018;27(1):276-286. [ CrossRef ] [ Medline ]
• Lindgren BM, Ringnér A, Molin J, Graneheim UH. Patients' experiences of isolation in psychiatric inpatient care: insights from a meta-ethnographic study. Int J Ment Health Nurs. 2019;28(1):7-21. [ CrossRef ] [ Medline ]
• Molin J, Graneheim UH, Ringnér A, Lindgren BM. From ideals to resignation—interprofessional teams perspectives on everyday life processes in psychiatric inpatient care. J Psychiatr Ment Health Nurs. 2016;23(9-10):595-604. [ CrossRef ] [ Medline ]
• Donald F, Duff C, Lee S, Kroschel J, Kulkarni J. Consumer perspectives on the therapeutic value of a psychiatric environment. J Ment Health. 2015;24(2):63-67. [ CrossRef ] [ Medline ]
• Stewart D, Burrow H, Duckworth A, Dhillon J, Fife S, Kelly S, et al. Thematic analysis of psychiatric patients' perceptions of nursing staff. Int J Ment Health Nurs. 2015;24(1):82-90. [ CrossRef ] [ Medline ]
• Molin J, Strömbäck M, Lundström M, Lindgren BM. It's not just in the walls: patient and staff experiences of a new spatial design for psychiatric inpatient care. Issues Ment Health Nurs. 2021;42(12):1114-1122. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lindgren BM, Molin J, Lundström M, Strömbäck M, Renberg ES, Ringnér A. Does a new spatial design in psychiatric inpatient care influence patients' and staff's perception of their care/working environment? A study protocol of a pilot study using a single-system experimental design. Pilot Feasibility Stud. 2018;4:191. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Laursen J, Danielsen A, Rosenberg J. Effects of environmental design on patient outcome: a systematic review. HERD. 2014;7(4):108-119. [ CrossRef ] [ Medline ]
• Johansson IM, Skärsäter I, Danielson E. The meaning of care on a locked acute psychiatric ward: patients' experiences. Nord J Psychiatry. 2009;63(6):501-507. [ CrossRef ] [ Medline ]
• Baumstarck K, Boyer L, Boucekine M, Aghababian V, Parola N, Lançon C, et al. Self-reported quality of life measure is reliable and valid in adult patients suffering from schizophrenia with executive impairment. Schizophr Res. 2013;147(1):58-67. [ CrossRef ] [ Medline ]
• Reininghaus U, Priebe S. Measuring patient-reported outcomes in psychosis: conceptual and methodological review. Br J Psychiatry. 2012;201(4):262-267. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fernandes S, Fond G, Zendjidjian X, Michel P, Baumstarck K, Lancon C, et al. The Patient-Reported Experience Measure for Improving qUality of care in Mental health (PREMIUM) project in France: study protocol for the development and implementation strategy. Patient Prefer Adherence. 2019;13:165-177. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schürhoff F, Fond G, Berna F, Bulzacka E, Vilain J, Capdevielle D, et al. A national network of schizophrenia expert centres: an innovative tool to bridge the research-practice gap. Eur Psychiatry. 2015;30(6):728-735. [ CrossRef ] [ Medline ]
• Dai S, Vo TT, Kehinde OJ, He H, Xue Y, Demir C, et al. Performance of polytomous IRT models with rating scale data: an investigation over sample size, instrument length, and missing data. Front Educ. 2021;6:721963. [ FREE Full text ] [ CrossRef ]
• Cappelleri JC, Lundy JJ, Hays RD. Overview of classical test theory and item response theory for the quantitative assessment of items in developing patient-reported outcomes measures. Clin Ther. 2014;36(5):648-662. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nguyen TH, Han HR, Kim MT, Chan KS. An introduction to item response theory for patient-reported outcome measurement. Patient. 2014;7(1):23-35. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ware J, Kosinski M, Keller S. How to score the SF-12 physical and mental health summary scales, 2nd edition. The Health Institute, New England Medical Center. Boston, MA.; 1995. URL: https:/​/www.​researchgate.net/​profile/​John-Ware-6/​publication/​291994160_How_to_score_SF-12_items/​links/​58dfc42f92851c369548e04e/​How-to-score-SF-12-items.​pdf [accessed 2023-02-07]
• Salyers MP, Bosworth HB, Swanson JW, Lamb-Pagone J, Osher FC. Reliability and validity of the SF-12 health survey among people with severe mental illness. Med Care. 2000;38(11):1141-1150. [ CrossRef ] [ Medline ]
• Embretson SE, Reise SP. Item Response Theory for Psychologists. Mahwah, NJ. Lawrence Erlbaum Associates; 2000.
• Reeve BB, Hays RD, Bjorner JB, Cook KF, Crane PK, Teresi JA, et al. Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Med Care. 2007;45(5 Suppl 1):S22-S31. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Muraki E. A generalized partial credit model: application of an EM algorithm. Appl Psychol Meas. 1992;16(2):159-176. [ CrossRef ]
• Masters GN. A rasch model for partial credit scoring. Psychometrika. 1982;47(2):149-174. [ CrossRef ]
• Bock RD, Aitkin M. Marginal maximum likelihood estimation of item parameters: application of an EM algorithm. Psychometrika. 1981;46(4):443-459. [ CrossRef ]
• Zieky M. Practical questions in the use of DIF statistics in test development. In: Wainer H, Holland PW, editors. Differential Item Functioning. Hillsdale, NJ. Lawrence Erlbaum Associates; 1993:337-347.
• Rogers HJ. Differential item functioning. In: Everitt BS, Howell DC, editors. Encyclopedia of Statistics in Behavioral Science. Chichester, UK. John Wiley & Sons; 2005:485-490.
• Chen H, Li M, Wang J, Xue C, Ding T, Nong X, et al. Factors influencing inpatients' satisfaction with hospitalization service in public hospitals in Shanghai, People's Republic of China. Patient Prefer Adherence. 2016;10:469-477. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Priebe S, Miglietta E. Assessment and determinants of patient satisfaction with mental health care. World Psychiatry. 2019;18(1):30-31. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Harvill LM. Standard error of measurement. Educ Meas Issues Pract. 1991;10(2):33-41. [ CrossRef ]
• Choi SW, Swartz RJ. Comparison of CAT item selection criteria for polytomous items. Appl Psychol Meas. 2009;33(6):419-440. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297-334. [ CrossRef ]
• Browne MW, Cudeck R. Alternative ways of assessing model fit. Sociol Methods Res. 1992;21(2):230-258. [ CrossRef ]
• Kline RB. Principles and Practice of Structural Equation Modeling, 2nd Edition. New York, NY. Guilford Press; 2005.
• Bjorner JB, Kosinski M, Ware JE. Calibration of an item pool for assessing the burden of headaches: an application of item response theory to the Headache Impact Test (HIT). Qual Life Res. 2003;12(8):913-933. [ CrossRef ] [ Medline ]
• Fliege H, Becker J, Walter OB, Bjorner JB, Klapp BF, Rose M. Development of a Computer-Adaptive Test for depression (D-CAT). Qual Life Res. 2005;14(10):2277-2291. [ CrossRef ] [ Medline ]
• Akaike H. A new look at the statistical model identification. IEEE Trans Automat Contr. 1974;19(6):716-723. [ CrossRef ]
• Schwarz G. Estimating the dimension of a model. Ann Statist. 1978;6(2):461-464. [ FREE Full text ] [ CrossRef ]
• Ware JE, Bjorner JB, Kosinski M. Practical implications of item response theory and computerized adaptive testing: a brief summary of ongoing studies of widely used headache impact scales. Med Care. 2000;38(Suppl 9):II73-I182. [ Medline ]
• Baker FB. The Basics of Item Response Theory, 2nd Edition. Washington, DC. ERIC Clearinghouse on Assessment and Evaluation; 2001.
• Chang HH, Ying Z. A global information approach to computerized adaptive testing. Appl Psychol Meas. 1996;20(3):213-229. [ CrossRef ]
• Bond TG, Fox CM. Applying the Rasch Model: Fundamental Measurement in the Human Sciences, 3rd Edition. New York, NY. Lawrence Erlbaum Associates; 2015.
• Wright B, Linacre JM. Reasonable mean-square fit values. Rasch Meas Trans. 1994;8:370-371. [ FREE Full text ]
• Zumbo BD. A handbook on the theory and methods of Differential Item Functioning (DIF): logistic regression modeling as a unitary framework for binary and likert-type (ordinal) item scores. Directorate of Human Resources Research and Evaluation, Department of National Defense. Ottawa, ON.; 1999. URL: https://faculty.educ.ubc.ca/zumbo/DIF/handbook.pdf [accessed 2023-02-20]
• Choi SW, Reise SP, Pilkonis PA, Hays RD, Cella D. Efficiency of static and computer adaptive short forms compared to full-length measures of depressive symptoms. Qual Life Res. 2010;19(1):125-136. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Michel P, Auquier P, Baumstarck K, Pelletier J, Loundou A, Ghattas B, et al. Development of a cross-cultural item bank for measuring quality of life related to mental health in multiple sclerosis patients. Qual Life Res. 2015;24(9):2261-2271. [ CrossRef ] [ Medline ]
• Rose M, Bjorner JB, Becker J, Fries JF, Ware JE. Evaluation of a preliminary physical function item bank supported the expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS). J Clin Epidemiol. 2008;61(1):17-33. [ CrossRef ] [ Medline ]
• Ferketich S. Focus on psychometrics. Aspects of item analysis. Res Nurs Health. 1991;14(2):165-168. [ CrossRef ] [ Medline ]
• Chalmers RP. Mirt: a Multidimensional Item Response Theory package for the R Environment. J Stat Softw. 2012;48(6):1-29. [ FREE Full text ] [ CrossRef ]
• Choi SW, Gibbons LE, Crane PK. Lordif: an R Package for detecting differential item functioning using iterative hybrid ordinal logistic regression/item response theory and Monte Carlo simulations. J Stat Softw. 2011;39(8):1-30. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dueber D. Package 'BifactorIndicesCalculator': a package for computing statistical indices relevant to bifactor measurement models. CRAN. 2022. URL: https://cran.r-project.org/web/packages/BifactorIndicesCalculator/BifactorIndicesCalculator.pdf [accessed 2023-03-16]
• Chalmers RP. Generating adaptive and non-adaptive test interfaces for multidimensional item response theory applications. J Stat Softw. 2016;71(5):1-38. [ FREE Full text ] [ CrossRef ]
• Elf M, Nordin S, Wijk H, Mckee KJ. A systematic review of the psychometric properties of instruments for assessing the quality of the physical environment in healthcare. J Adv Nurs. 2017;73(12):2796-2816. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Reise SP, Scheines R, Widaman KF, Haviland MG. Multidimensionality and structural coefficient bias in structural equation modeling: a bifactor perspective. Educ Psychol Meas. 2012;73(1):5-26. [ CrossRef ]
• Cook KF, Kallen MA, Amtmann D. Having a fit: impact of number of items and distribution of data on traditional criteria for assessing IRT's unidimensionality assumption. Qual Life Res. 2009;18(4):447-460. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kwan YH, Uy EJ, Bautista DC, Xin X, Xiao Y, Lee GL, et al. Development and calibration of a novel positive mindset item bank to measure Health-Related Quality of Life (HRQoL) in Singapore. PLoS One. 2019;14(7):e0220293. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Haley SM, Fragala-Pinkham MA, Dumas HM, Ni P, Gorton GE, Watson K, et al. Evaluation of an item bank for a computerized adaptive test of activity in children with cerebral palsy. Phys Ther. 2009;89(6):589-600. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Crins MHP, Terwee CB, Klausch T, Smits N, de Vet HCW, Westhovens R, et al. The Dutch-Flemish PROMIS physical function item bank exhibited strong psychometric properties in patients with chronic pain. J Clin Epidemiol. 2017;87:47-58. [ CrossRef ] [ Medline ]
• Pilkonis PA, Yu L, Dodds NE, Johnston KL, Lawrence SM, Hilton TF, et al. An item bank for abuse of prescription pain medication from the Patient-Reported Outcomes Measurement Information System (PROMIS®). Pain Med. 2017;18(8):1516-1527. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rindestig FC, Wiberg M, Chaplin JE, Henje E, Dennhag I. Psychometrics of three Swedish physical pediatric item banks from the Patient-Reported Outcomes Measurement Information System (PROMIS)®: pain interference, fatigue, and physical activity. J Patient Rep Outcomes. 2021;5(1):105. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Terwee CB, Crins MHP, Boers M, de Vet HCW, Roorda LD. Validation of two PROMIS item banks for measuring social participation in the Dutch general population. Qual Life Res. 2019;28(1):211-220. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kandel H, Khadka J, Goggin M, Pesudovs K. Patient-reported outcomes for assessment of quality of life in refractive error: a systematic review. Optom Vis Sci. 2017;94(12):1102-1119. [ CrossRef ] [ Medline ]
• Olusina AK, Ohaeri JU, Olatawura MO. Patient and staff satisfaction with the quality of in-patient psychiatric care in a Nigerian general hospital. Soc Psychiatry Psychiatr Epidemiol. 2002;37(6):283-288. [ CrossRef ] [ Medline ]
• Lundqvist LO, Ahlström G, Wilde-Larsson B, Schröder A. The patient's view of quality in psychiatric outpatient care: patients' view of psychiatric care. J Psychiatr Ment Health Nurs. 2012;19(7):629-637. [ CrossRef ] [ Medline ]
• Beckers T, Koekkoek B, Tiemens B, Jaeqx-van Tienen L, Hutschemaekers G. Substituting specialist care for patients with severe mental illness with primary healthcare. Experiences in a mixed methods study. J Psychiatr Ment Health Nurs. 2019;26(1-2):1-10. [ CrossRef ] [ Medline ]
• Zendjidjian XY, Auquier P, Lançon C, Loundou A, Parola N, Faugère M, et al. Determinants of patient satisfaction with hospital health care in psychiatry: results based on the SATISPSY-22 questionnaire. Patient Prefer Adherence. 2014;8:1457-1464. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Roche E, Madigan K, Lyne JP, Feeney L, O'Donoghue B. The therapeutic relationship after psychiatric admission. J Nerv Ment Dis. 2014;202(3):186-192. [ CrossRef ] [ Medline ]
• Wyder M, Bland R, Blythe A, Matarasso B, Crompton D. Therapeutic relationships and involuntary treatment orders: service users' interactions with health-care professionals on the ward. Int J Ment Health Nurs. 2015;24(2):181-189. [ CrossRef ] [ Medline ]
• Fond G, Young AH, Godin O, Messiaen M, Lançon C, Auquier P, et al. Improving diet for psychiatric patients: high potential benefits and evidence for safety. J Affect Disord. 2020;265:567-569. [ CrossRef ] [ Medline ]
• Gill R, Tyndall SF, Vora D, Hasan R, Megna JL, Leontieva L. Diet quality and mental health amongst acute inpatient psychiatric patients. Cureus. 2021;13(1):e12434. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Messina G, Fenucci R, Vencia F, Niccolini F, Quercioli C, Nante N. Patients' evaluation of hospital foodservice quality in Italy: what do patients really value? Public Health Nutr. 2013;16(4):730-737. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Scott NW, Fayers PM, Aaronson NK, Bottomley A, de Graeff A, Groenvold M, et al. Differential Item Functioning (DIF) analyses of health-related quality of life instruments using logistic regression. Health Qual Life Outcomes. 2010;8:81. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hagan TL, Belcher SM, Donovan HS. Mind the mode: differences in paper vs. web-based survey modes among women with cancer. J Pain Symptom Manage. 2017;54(3):368-375. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by J Torous; submitted 13.06.23; peer-reviewed by M Kamouchi, D Dinh, L Hua; comments to author 11.08.23; revised version received 15.09.23; accepted 21.01.24; published 16.05.24.

©Sara Fernandes, Yann Brousse, Xavier Zendjidjian, Delphine Cano, Jérémie Riedberger, Pierre-Michel Llorca, Ludovic Samalin, Daniel Dassa, Christian Trichard, Vincent Laprevote, Anne Sauvaget, Mocrane Abbar, David Misdrahi, Fabrice Berna, Christophe Lancon, Nathalie Coulon, Wissam El-Hage, Pierre-Emmanuel Rozier, Michel Benoit, Bruno Giordana, Alejandra Caqueo-Urízar, Dong Keon Yon, Bach Tran, Pascal Auquier, Guillaume Fond, Laurent Boyer. Originally published in JMIR Mental Health (https://mental.jmir.org), 16.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.

Alcohol’s impact on the health and wellbeing of women in low- and middle-income countries: An integrative review

• Anne-Marie Laslett, Ph.D. Centre for Alcohol Policy Research, La Trobe University, Melbourne, Australia; National Drug Research Institute, Curtin University, Perth, Australia
• Megan Cook Centre for Alcohol Policy Research, La Trobe University, Melbourne, Australia; Institute for Social Marketing & Health, University of Stirling, Stirling, United Kingdom.
• Leane Ramsoomar South African Medical Research Council, Pretoria, South Africa
• Neo Morojele Psychology, University of Johannesburg
• Orratai Waleewong International Health Policy Program, Ministry of Health, Thailand

In lower- and middle-income countries (LMICs), alcohol poses significant challenges to the health, well-being, and safety of women. This integrative review builds upon a previous rapid assessment conducted by Laslett and Cook in 2019, delving into the specific impacts of alcohol on women's health in LMICs while also shedding light on research gaps across different global regions. We advocate for further research endeavours and policy initiatives aimed at addressing these deficiencies. Additionally, we prompt discussions regarding the necessity of establishing distinct priorities for alcohol research, monitoring, policy, and evaluation tailored to women's needs in LMICs, as compared to those for men or the general population, particularly in high-income countries (HICs).

Author Biography

Anne-Marie Laslett is a Senior Research Fellow at the National Drug Research Institute, Curtin University focussing on alcohol’s harms to children. She directs the Range and Magnitude of Alcohol’s Harm to Others project at the Centre for Alcohol Policy Research, La Trobe University. For twenty years she has undertaken alcohol and other drug research, focusing latterly on alcohol’s harms to children and families. She originally graduated as a dentist, and then undertook Masters Degrees in Oral Epidemiology and Public Health and a PhD in population and global health.

• DOWNLOAD FULL PAPER
• Supplementary Material
• Endnote/Zotero/Mendeley (RIS)

In consideration of publishing this article the authors transfer, assign, or otherwise convey all copyright ownership to the International Journal of Alcohol and Drug Research. By this transfer, the article becomes the property of the International Journal of Alcohol and Drug Research and may not be published elsewhere without written permission from the journal.

This transfer of copyright also implies transfer of rights for printed, electronic, microfilm, and facsimile publication. The author(s) will receive no royalty or other monetary compensation for transferring the copyright of the article to the International Journal of Alcohol and Drug Research . IJADR, in turn, grants each author the right to republish the article, without paying royalties to IJADR, in any book of which he or she is the author or editor, subject to the express conditions that (a) the author notify the International Journal of Alcohol and Drug Research in writing of this republication and (b) a credit line attributes the original publication to the International Journal Of Alcohol and Drug Research .

Articles are licenced with a Creative Commons License Deed  -- you are free to share articles but must give appropriate attribution, may not use for commercial purposes or distribute modified works. See CC/BY-NC/ND/4.0/ .

Author Agreement:

As the submitting author, and on behalf of all of the manuscript authors I agree with the terms above relating to the copyright transfer of the manuscript to the International Journal of Alcohol and Drug Research.

• • Alcohol Prevention Research and Policy Development in LMICs: Facilitating Partnerships, Capacity and Impact
• • KBS Conference 2023 Issue
• For Readers
• For Authors
• For Librarians

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• News & Views
• David Oliver: The risk...

David Oliver: The risk assessment for NHS England’s long term workforce plan should worry doctors

Linked news.

NHS England is accused of complacency over workforce plan risk assessment

• Related content
• Peer review
• David Oliver , consultant in geriatrics and acute general medicine
• davidoliver372{at}googlemail.com Follow David on Twitter @mancunianmedic

In June 2023 NHS England (NHSE) published its long awaited and serially delayed long term workforce plan for the NHS. 1 In the 10 months since then we’ve seen pay disputes, difficult contract negotiations, strike action, and ballots of clinical staff. 2 3 4 While the BMA’s Consultants Committee has now settled a long running contract dispute with the government, 5 its Junior Doctors Committee seems no closer to a negotiated settlement. 6 We continue to face serious problems with workforce retention and morale, as well as staffing and rota gaps across a range of clinical disciplines—in nursing even more so than medicine. 7

Last month the BMA published a paper on “tackling medical attrition,” 8 estimating that in 2022-23 some 15 000 to 23 000 doctors had left the NHS workforce, that replacing them would cost £1.6bn to £2.4bn, and that ever more doctors were taking steps to leave. It called for urgent action on retention through a variety of practical actions.

There’s also been considerable controversy over the expansion of physician associate and anaesthesia associate roles announced in the workforce plan and over the General Medical Council’s plans for their regulation. 9 10 This has drawn several medical royal colleges into fractious disputes with their own members. 11 12 13

FOI request

One would hope that NHSE had anticipated some of these troubles and hadn’t been complacent about mitigating or preventing them. I wanted to understand what attempts it had made to anticipate and plan for such problems as part of developing its workforce plan and whether it had tried to modify or avoid serious risks. So, in February I sent NHSE a freedom of information request asking if it could share the risk assessment that had accompanied the workforce plan. My requests were met openly, with only names of individual officials redacted. 14

For such a major policy initiative with such serious long term implications, I was surprised by just how scant the risk assessment was. It contained only 12 areas of risk: these included the timeline for delivery; the fact that the NHS service strategy was in development and so would change workforce needs; that national policy decisions, such as around terms and conditions, could “destabilise industrial relations”; and that “multiple layers of governance” involving multiple organisations might hamper planning and delivery. Industrial relations and the impact of national policy decisions continue to play out, and the other issues remain very real.

The original workforce plan had contained a commitment to “adequate growth in foundation placement capacity” and “a commensurate increase in specialty training places that meets the demands of the NHS of the future,” while pledging “to work with stakeholders to ensure this growth is sustainable and focused in the service areas where need is greatest.” Yet these areas lack operational detail in the plan, and right now we have serious concerns about foundation doctors being sent to any part of the country, at short notice with little say in the matter, or finding themselves temporarily unemployed. 15 16 Also worrying is that doctors entering core or higher specialty training, or indeed completing their postgraduate training (for instance, as GPs) are stuck at bottlenecks and unable to find substantive, stable work. 17

There’s currently no financial commitment and no detailed logistical delivery plan or timeline for delivering the aims of the workforce plan. For instance, is there any funding to pay for the workforce or medical and nursing school expansion or capital investment in facilities? There have already been news reports of a pause in the expansion of medical school places, especially for UK citizens, with newer medical schools complaining about the lack of support. 18

Regarding any impact assessment of the growth of various workforce groups including physician and anaesthesia associates on the rest of the workforce or on training, the official answer I received was that “no specific risk assessments were undertaken at programme level.” In view of subsequent events this seems overly complacent.

Watering down

I completely understand the political constraints on a plan whose intended life will traverse at least three parliaments, several spending reviews, and changing economic circumstances. But I have serious doubts about the rigour of the risk assessment, the lack of impact assessment, and the failure to anticipate the backlash against some of the plan’s elements. There’s no cross party commitment to the plan, which puts its delivery at risk after the next general election. Delivery certainly doesn’t seem to be on schedule currently.

What we can hope for is that sufficiently good governance, learning, engagement, and review structures and mechanisms can be put in place so that the workforce plan can be updated and improved as it’s implemented, to mitigate some of the risks currently unfolding. However, if our economic performance as a nation continues to decline, I can envisage further watering down or delayed timelines for the headline commitments.

Competing interests: See bmj.com/about-bmj/freelance-contributors

Provenance and peer review: Commissioned; not externally peer reviewed.

• ↵ NHS England. NHS long term workforce plan. Updated 22 Apr 2024. https://www.england.nhs.uk/publication/nhs-long-term-workforce-plan/
• ↵ Government and health unions agree pay deal paving way for an end to strike action - GOV https://www.gov.uk/government/news/government-and-health-unions-agree-pay-deal-paving-way-for-an-end-to-strike-action
• ↵ Junior doctors vote for strikes in England over NHS pay - BBC News. https://www.bbc.co.uk/news/health-64653488
• ↵ Consultants to ballot for industrial action on pay and pensions. https://www.bma.org.uk/news-and-opinion/consultants-to-ballot-for-industrial-action-on-pay-and-pensions
• ↵ NHS consultants accept pay offer, ending year-long dispute with government | NHS | The Guardian https://www.theguardian.com/society/2024/apr/05/nhs-consultants-accept-government-pay-offer-bma-end-strike-dispute
• ↵ Pay restoration for junior doctors in England. https://www.bma.org.uk/our-campaigns/junior-doctor-campaigns/pay/pay-restoration-for-junior-doctors-in-england
• ↵ Waste not, want not. https://www.nuffieldtrust.org.uk/sites/default/files/2023-09/NuffieldTrust-Wastenotwantnot_WEB_FINAL.pdf
• ↵ Tackling medical attrition in the UK’s health services. https://www.bma.org.uk/attrition-costs
• ↵ Why the fuss about physician associates? | The BMJ https://www.bmj.com/content/385/bmj.q862.full
• ↵ Physician associates. Doctors raise alarm over legislation to allow GMC regulation | The BMJ https://www.bmj.com/content/384/bmj.q156
• ↵ Royal College of Physicians announces independent review among next steps post EGM | RCP London https://www.rcp.ac.uk/news-and-media/news-and-opinion/royal-college-of-physicians-announces-independent-review-among-next-steps-post-egm/
• Physician associates
• ↵ RCoA position statement on anaesthesia associates | The Royal College of Anaesthetists https://www.rcoa.ac.uk/news/rcoa-position-statement-anaesthesia-associates
• ↵ New foundation doctors express anger at being allocated “placeholder job” | The BMJ https://www.bmj.com/content/385/bmj.q953
• ↵ Placeholder jobs” are isolating doctors at the start of their careers | The BMJ https://www.bmj.com/content/385/bmj.q1056
• ↵ The growing bottlenecks in specialty training | The BMJ https://www.bmj.com/content/382/bmj.p1732
• ↵ Government delays plans to double number of medical students in England | NHS | The Guardian https://www.theguardian.com/society/2024/feb/25/government-u-turn-on-plans-to-double-number-of-medical-students-in-england

YSE Class of ’24: Aishwarya Iyer Focuses on Underrepresented Voices in Energy Usage

By Jen DeMoss

When Aishwarya Iyer ’24 PhD was in the eighth grade in Maharashtra, India, she performed a home energy audit for a science competition that involved assessing residents’ energy usage and making suggestions for maximizing energy efficiency. Even in that early examination of residential energy usage and preferences, Iyer was struck by how small adjustments could avert carbon emissions.

Commencement 2024 Video and News Hub

Full coverage of Commencement 2024, including the highlights video, photo gallery, graduate spotlights, and the YSE ceremony livestream video.

“The project led me deep into studying climate change,” Iyer recalled. “I wasn’t familiar with the concept. Afterwards, it was my chosen field.”

She earned her bachelor’s and master’s degrees in energy science and engineering at the Indian Institute of Technology (IIT) in Bombay, where she examined the intersection of energy and the environment. Her field work included assessing energy usage within IIT campus buildings as well as for the Canadian Cement Sector.

At the Yale School of the Environment, Iyer pursued her interest in energy and the built environment at the Center for Industrial Ecology , focusing her research on the life cycle assessment of energy use and the environmental impacts of residential buildings in India. As she gathered the information, she realized that informal and low-income housing units were not represented in the data.

“During this process, I found that all the available data came from formal, multi-story buildings, even though many urban residents live in informal homes,” she said. “That immediately struck me as a gap in our understanding of energy usage.”

To address this issue, she devised a study that would include low-income and informal housing units in an assessment of home cooling usage in relation to rising temperatures and humidity in urban areas. While some research predicts that demands for home cooling and energy will skyrocket as temperatures rise, Iyer examined whether mitigating factors — including behavior of residents in the underrepresented communities — could lower energy usage estimates. She noted that in her experience, residents often supplement air conditioning by using natural ventilation, sometimes the only affordable way to cool a dwelling. She collected data in India on building floor plans, construction materials, and other metrics, and used energy modeling and architectural software to test energy use scenarios and temperature and humidity variables.

I’ve met such a diverse group of people. I’ve learned to think about the importance of human behavior, well-being, and research representation. My experience has been a researcher’s dream.” 

Her findings suggest that India will require far less energy than predicted to cool its growing urban populations and that building density and type greatly influence thermal comfort.

“Air conditioning may not be as necessary as some expect. We might be able to mitigate more than 50% of the estimated energy demand by incorporating practices like opening windows and understanding that people in India are comfortable at higher temperatures,” Iyer said.

Iyer’s work revealed the diversity of informal building types and the influence of spatial density and building materials on heat stress, said Narasimha Rao, associate professor of energy systems, who helped advise her research.

“This work is important to improve the representation of building stocks in the Global South and their energy-demand projections, which are a major driver of future global greenhouse gas emissions,” Rao said. “It shows that current projections of building energy demand growth are based on more simplistic representations and may overestimate demand significantly.”

While at YSE, Iyer served as  vice president of the Yale Graduate Student Consulting Club and Chair of G ordon Research Conferences . She also spent two semesters as a visiting researcher at the Norwegian University of Science and Technology.

Subscribe to “YSE 3”

Biweekly, we highlight three news and research stories about the work we’re doing at Yale School of the Environment.

Iyer said that the interdisciplinary nature of YSE  gave her the freedom to pursue diverse coursework and collaborations. Her doctoral committee includes Anna Dyson, Hines Professor of Sustainable Architectural Design, and she  participated in the United Nations Environment Programme project with School of Architecture associates. 

“One of the most powerful impacts for me, because my research spans energy and environment, was my program’s openness to interdisciplinarity,” Iyer said. 

She  also said the breadth of her graduate experience has  prepared her for her next step  — working as a consultant with Boston Consulting Group.

“As an engineer, I was trained to focus on numbers and the energy side of things,” she said. “But working in Professor Rao’s lab, I’ve met such a diverse group of people. I’ve learned to think about the importance of human behavior, well-being, and research representation. My experience has been a researcher’s dream.”

Class of ’24

YSE Class of ’24: Jane Jacoby Is at Home at the Intersection of Environmental Law and Justice

YSE Class of ’24: Sangam Paudel Seeks to Balance Conservation with Economic Needs

• Doctoral Program
• Profiles / Features

Media Contact

Fran Silverman

Associate Director of Communications

Students in the News

Unique Research on Calving Impacts on Nutrient Cycle Earns 2024 Bormann Prize

Connect with us.

• Request Information
• Register for Events

BRIEF RESEARCH REPORT article

This article is part of the research topic.

Industry 4.0: The Impact and Implications for Transitioning Process Industries

Matchmaking for Industrial Symbiosis: a digital tool for the identification, quantification and optimisation of symbiotic potential in Industrial Ecosystems Provisionally Accepted

• 1 National Technical University of Athens, Greece

The final, formatted version of the article will be published soon.

Effective waste management is crucial for sustainable industrial operations. This paper introduces a state-of-the-art digital tool designed for the circular economy. Primarily it pinpoints and quantifies symbiotic possibilities between industries with liquid waste streams, emphasising the most lucrative inter-industry connections. In practice, the tool takes in data such as waste stream volumes, material concentrations within these streams, market prices of materials, and industries' raw material consumption rates. Utilising these, its algorithm identifies and assesses the most profitable material exchanges among the specified industries. This assessment considers the market value of materials and the costs associated with recovering those materials from liquid waste streams. One of the major challenges, the estimation of recovery costs, is addressed using an innovative Sherwood plot analysis. This analysis draws a correlation between a material's recovery cost and its concentration within a liquid medium. The tool's output provides a detailed list of potential transactions complemented by illustrative graphs that detail mass flows, profit margins, and environmental advantages for each industry. Collectively, these details offer insights both for individual industries and the industrial ecosystem as a whole. One of the tool's most significant revelations is its ability to uncover potential 'bridges' linking industrial waste streams to resource needs, unearthing previously unnoticed economic and ecological gains. By calculating economic and environmental benefits of "waste" reuse, this tool offers a compelling rationale for the adoption of industrial symbiosis. Ultimately, it uncovers the transformative potential of aligning industrial activities with a balance that fosters both economic growth and ecological responsibility.

Keywords: industrial symbiosis 1, circular economy 2, sustainability 3, material reclamation 4, waste valorisation 5, digital tool 6, optimisation 7, Sherwood plot 8 . (Min.5-Max. 8)

Received: 31 Dec 2023; Accepted: 16 May 2024.

Copyright: © 2024 Makropoulos, Kritikos and Pantazis. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Mr. Nikolaos Alexandros Kritikos, National Technical University of Athens, Athens, 15780, Greece

People also looked at

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.324(7330); 2002 Jan 19

Learning needs assessment: assessing the need

Learning needs assessment has a fundamental role in education and training, but care is needed to prevent it becoming a straitjacket

It might seem self evident that the need to learn should underpin any educational system. Indeed, the literature suggests that, at least in relation to continuing professional development, learning is more likely to lead to change in practice when needs assessment has been conducted, the education is linked to practice, personal incentive drives the educational effort, and there is some reinforcement of the learning. 1 Learning needs assessment is thus crucial in the educational process, but perhaps more of this already occurs in medical education than we suspect. The key lesson might be for those who design new systems of education and training: for example, the postgraduate education allowance system in general practice was felt to fail the profession because it did not include needs assessment and so led to ad hoc education to fulfil the time requirements of the system rather than the needs of individual doctors or the profession as a whole. On the other hand, basing learning in a profession entirely on the assessment of needs is a dangerous and limiting tactic. So a balance must be struck.

Summary points

• Learning needs assessment is a crucial stage in the educational process that leads to changes in practice, and has become part of government policy for continuing professional development
• Learning needs assessment can be undertaken for many reasons, so its purpose should be defined and should determine the method used and the use made of findings
• Exclusive reliance on formal needs assessment could render education an instrumental and narrow process rather than a creative, professional one
• Different learning methods tend to suit different doctors and different identified learning needs
• Doctors already use a wide range of formal and informal ways of identifying their own learning needs as part of their ordinary practice
• These should be the starting point in designing formalised educational systems for professional improvement

Learning needs assessment in medicine

In 1998 both individual and organisational needs assessment became part of government policy in relation to the continuing professional development and personal development plans of all healthcare professionals. 2 Thus, it has a role in the clinical governance of the service 3 and is therefore much more than an educational undertaking. This integration of needs assessment, education, and quality assurance of the service was first made explicit in 1989 in relation to clinical audit, which would identify practices in need of improvement and ensure that educational and organisational interventions were made to address these needs. 4 Accordingly, audit was described as “essentially educational” and the educational process surrounding it described. 5

Long before these recent developments, needs assessment outside medicine was presented as an important part of managed education and learning contracts, which are the predecessors of the personal development plans to be developed for all NHS healthcare professionals. 6 In his descriptions of adult learning Knowles assumed (he did not claim to have research evidence) that learners needed to feel a necessity to learn and that identifying one's own learning needs was an essential part of self directed learning. 7 In medicine a doctor's motivation to learn would therefore derive from needs identified during his or her experience of clinical practice. So the pedigree and practice of learning needs assessment, if not the evidence, are well established.

The definition of need

As in most areas of education, for many years there has been intense debate about the definition, purpose, validity, and methods of learning needs assessment. 8 It might be to help curriculum planning, diagnose individual problems, assess student learning, demonstrate accountability, improve practice and safety, or offer individual feedback and educational intervention. Published classifications include felt needs (what people say they need), expressed needs (expressed in action) normative needs (defined by experts), and comparative needs (group comparison). 9 Other distinctions include individual versus organisational or group needs, clinical versus administrative needs, and subjective versus objectively measured needs. 10 The defined purpose of the needs assessment should determine the method used and the use made of findings.

Furthermore, even though the concept of educational needs assessment is enshrined in practice, policy, and the educational canon, several factors indicate the need for careful planning and research in this subject (see boxes ​ boxesB1 B1 and ​ andB2). B2 ). Exclusive reliance on formal needs assessment in educational planning could render education an instrumental and narrow process rather than a creative, professional one. This is especially so in a profession where there is inherent unpredictability and uncertainty. Members of any profession require wide knowledge and depth of experience—the relevance of some of which might not have been obvious at the time of learning. Certainly, learning needs can and should be identified on the basis of what has been experienced and of what more experienced members of the profession know to be relevant, but this must not deter other, more general or even speculative, learning that, at the time, seems to answer no specific need. Possibly no specific learning needs assessment would ever send a person to a large international conference on a generic subject (such as endocrinology, medical education, or management). It is, nevertheless, important that doctors attend such meetings and return with the unexpected and expected benefits that they accrue.

: Need for careful planning in needs assessment

• There is little evidence that needs assessment alone enhances educational effectiveness and outcomes, so it must be placed within the wider process of planned learning, relevance to practice, and reinforcement of learning in the appropriate context
• Formal needs assessment can identify only a narrow range of needs and might miss needs not looked for, so breadth and flexibility of needs assessment methods should be embraced
• In professional education it is not necessarily defensible to focus all learning on identified needs—wider professional learning not related to a specific need is also of fundamental value where practice is not predictable
• Individual and group learning needs are different—group learning needs may produce an average picture that fails to address important needs and interests of individual members of the group—so a balance is required. Each approach has its uses and effects, but each must be used for the right purpose
• Identifying individual learning needs, often not shared by others, may lead to an unimpressive cost-benefit analysis in terms of individually targeted use of educational resources if used inappropriately. Individual learning needs assessment is best used in the context of learning that occurs on an individual basis—such as in the relationship between general practitioner registrar and trainer

: Need for research into needs assessment in medical education

• What are the effects of and responses to needs assessment alone for students, trainees, and senior doctors at different stages of medical education?
• What is the relative validity, reliability, or utility of different formal and informal methods of learning needs assessment in medical education at any level?
• To what extent do needs assessment methods identify all important learning needs?
• What are the relative effects and efficacy of identifying group and individual learning needs?
• What methods of planning effective learning experiences are most effective on the basis of needs identified?

Methods of needs assessment

Although the literature generally reports only on the more formal methods of needs assessment, doctors use a wide range of informal ways of identifying learning needs as part of their ordinary practice. These should not be undervalued simply because they do not resemble research. Questionnaires and structured interviews seem to be the most commonly reported methods of needs assessment, but such methods are also used for evaluation, assessment, management, education, and now appraisal and revalidation. 11 Together, these formal and informal methods might make an effective battery where there is clarity of purpose. The Good CPD Guide details 46 formal and informal methods of self assessment (see box ​ boxB3 B3 ). 12

: Good CPD Guide 's classification of sources of needs assessment 12

The methods listed are both formal and informal, planned and opportunistic, showing that day to day work and encounters have the potential to generate needs as much as do formal methods. Formal needs assessment methods include critical incident techniques, gap analysis, objective knowledge and skills tests, observation, revalidation, self assessment, video assessment, and peer review. Such methods are often used to identify group needs. 14 , 15 Formal identification of needs can also arise from audit, morbidity patterns, adverse events, patient satisfaction surveys, and risk assessment. Most of these tools use quantitative methods that can generate computerised data and cover wider population ranges, but these are often unable to probe into the personal agendas and opinions of individuals.

Types of needs assessment

Methods of needs assessment can be classified into seven main types, each of which can take many different forms in practice.

Gap or discrepancy analysis

This formal method involves comparing performance with stated intended competencies—by self assessment, peer assessment, or objective testing—and planning education accordingly. 9 , 16 , 17

Reflection on action and reflection in action

Reflection on action is an aspect of experiential learning and involves thinking back to some performance, with or without triggers (such as videotape or audiotape), and identifying what was done well and what could have been done better. 18 , 19 The latter category indicates learning needs.

Reflection in action involves thinking about actual performance at the time that it occurs and requires some means of recording identified strengths and weaknesses at the time. The Canadian MOCOMP programme uses formalised reflection as its basic process. 20 Similarly, PUNs and DENs (see box ​ boxB3) B3 ) are well known in British general practice.

Self assessment by diaries, journals, log books, weekly reviews

This is an extension of reflection that involves keeping a diary or other account of experiences. 21 However, practice might show that such documents tend to be written nearer the time of their review than the time of the activity being recorded.

Peer review

This is rapidly becoming a favourite method. It involves doctors assessing each other's practice and giving feedback and perhaps advice about possible education, training, or organisational strategies to improve performance. The Good CPD Guide describes five types of peer review—internal, external, informal, multidisciplinary, and physician assessment. 11 The last of these is the most formal, involving rating forms completed by nominated colleagues, and shows encouraging levels of validity, reliability, and acceptability. 22 , 23

Observation

In more formal settings doctors can be observed performing specific tasks that can be rated by an observer, either according to known criteria or more informally. The results are discussed, and learning needs are identified. The observer can be a peer, a senior, or a disinterested person if the ratings are sufficiently objective or overlap with the observer's area of expertise (such as communication skills or management).

Critical incident review and significant event auditing

Although this technique is usually used to identify the competencies of a profession or for quality assurance, it can also be used on an individual basis to identify learning needs. 24 The method involves individuals identifying and recording, say, one incident each week in which they feel they should have performed better, analysing the incident by its setting, exactly what occurred, and the outcome and why it was ineffective.

Practice review

A routine review of notes, charts, prescribing, letters, requests, etc, can identify learning needs, especially if the format of looking at what is satisfactory and what leaves room for improvement is followed.

The difference between needs assessment and assessment

Needs assessment is not the same as assessment in the sense of examination of learning. Assessment systems that lead to academic or professional awards should show certain minimum characteristics, including measurement of performance against external criteria and standards, a decision on adequacy by an assessor, and standardised data gathering. 25 Needs assessment might sometimes have these characteristics, but it also might be based on practice, reflection, professional judgment, discussion, and informal data. Needs assessment methods that are limited by the standards of assessment will fall into the trap of assessing only a narrow range of needs.

Learning for needs

The main purpose of needs assessment must be to help educational planning, but this must not lead to too narrow a vision of learning. Learning in a profession is unlike any other kind of learning. Doctors live in a rich learning environment, constantly involved in and surrounded by professional interaction and conversation, educational events, information, and feedback. The search for the one best or “right” way of learning is a hopeless task, 1 especially if this is combined with attempting to “measure” observable learning. Research papers show, at best, the complexity of the process.

Multiple interventions targeted at specific behaviour result in positive change in that behaviour. 26 Exactly what those interventions are is less important than their multiplicity and targeted nature. On the other hand, different doctors use different learning methods to meet their individual needs. For example, in a study of 366 primary care doctors who identified recent clinical problems for which they needed more knowledge or skill to solve, 55 different learning methods were selected. 27 The type of problem turned out to be the major determinant of the learning method chosen, so there may not be one educational solution to identified needs.

Much of doctors' learning is integrated with their practice and arises from it. The style of integrated practice and learning (“situated learning”) develops during the successive stages of medical education. 28 The components of apprenticeship learning in postgraduate training are made up of many activities that may be regarded as part of practice (see box ​ boxB4 B4 ). 29 Senior doctors might also recognise much of their learning in some of these elements and could certainly add more—such as conversations with colleagues.

: Components of apprenticeship learning in postgraduate training 29

• Learning by doing
• Experience of seeing patients
• Building up personal knowledge and experience
• Discussing patients
• Managing patients
• Having errors corrected
• Making teaching points during service
• Listening to experts' explanations
• “Picking things up”
• Charismatic influences
• Learning clinical methods from practice
• Being questioned about thought and actions about patients
• Teaching by doing
• Using knowledge and skill
• Bite-size learning from “bits and pieces”
• Retrieving and applying knowledge stored in memory
• Learning from supervision
• Receiving feedback
• Presentation and summarising
• Observing experts working
• Learning from role models
• Learning from team interactions
• Hearing consultants thinking aloud
• Thinking about practice and patients

Thus, educational planning on the basis of identified needs faces real challenges in making learning appropriate to and integrated with professional style and practice. The first step in all of this is to recognise the needs assessment and learning that are a part of daily professional life in medicine and to formalise, highlight, and use these as the basis of future recorded needs assessment and subsequent planning and action, as well as integrating them with more formal methods of needs assessment to form a routine part of training, learning, and improving practice.

Editorial by Goldbeck-Wood and Peile

  Competing interests: None declared.

McKinsey Global Private Markets Review 2024: Private markets in a slower era

At a glance, macroeconomic challenges continued.

McKinsey Global Private Markets Review 2024: Private markets: A slower era

If 2022 was a tale of two halves, with robust fundraising and deal activity in the first six months followed by a slowdown in the second half, then 2023 might be considered a tale of one whole. Macroeconomic headwinds persisted throughout the year, with rising financing costs, and an uncertain growth outlook taking a toll on private markets. Full-year fundraising continued to decline from 2021’s lofty peak, weighed down by the “denominator effect” that persisted in part due to a less active deal market. Managers largely held onto assets to avoid selling in a lower-multiple environment, fueling an activity-dampening cycle in which distribution-starved limited partners (LPs) reined in new commitments.

About the authors

This article is a summary of a larger report, available as a PDF, that is a collaborative effort by Fredrik Dahlqvist , Alastair Green , Paul Maia, Alexandra Nee , David Quigley , Aditya Sanghvi , Connor Mangan, John Spivey, Rahel Schneider, and Brian Vickery , representing views from McKinsey’s Private Equity & Principal Investors Practice.

Performance in most private asset classes remained below historical averages for a second consecutive year. Decade-long tailwinds from low and falling interest rates and consistently expanding multiples seem to be things of the past. As private market managers look to boost performance in this new era of investing, a deeper focus on revenue growth and margin expansion will be needed now more than ever.

Perspectives on a slower era in private markets

Global fundraising contracted.

Fundraising fell 22 percent across private market asset classes globally to just over $1 trillion, as of year-end reported data—the lowest total since 2017. Fundraising in North America, a rare bright spot in 2022, declined in line with global totals, while in Europe, fundraising proved most resilient, falling just 3 percent. In Asia, fundraising fell precipitously and now sits 72 percent below the region’s 2018 peak.

Despite difficult fundraising conditions, headwinds did not affect all strategies or managers equally. Private equity (PE) buyout strategies posted their best fundraising year ever, and larger managers and vehicles also fared well, continuing the prior year’s trend toward greater fundraising concentration.

The numerator effect persisted

Despite a marked recovery in the denominator—the 1,000 largest US retirement funds grew 7 percent in the year ending September 2023, after falling 14 percent the prior year, for example 1 “U.S. retirement plans recover half of 2022 losses amid no-show recession,” Pensions and Investments , February 12, 2024. —many LPs remain overexposed to private markets relative to their target allocations. LPs started 2023 overweight: according to analysis from CEM Benchmarking, average allocations across PE, infrastructure, and real estate were at or above target allocations as of the beginning of the year. And the numerator grew throughout the year, as a lack of exits and rebounding valuations drove net asset values (NAVs) higher. While not all LPs strictly follow asset allocation targets, our analysis in partnership with global private markets firm StepStone Group suggests that an overallocation of just one percentage point can reduce planned commitments by as much as 10 to 12 percent per year for five years or more.

Despite these headwinds, recent surveys indicate that LPs remain broadly committed to private markets. In fact, the majority plan to maintain or increase allocations over the medium to long term.

Investors fled to known names and larger funds

Fundraising concentration reached its highest level in over a decade, as investors continued to shift new commitments in favor of the largest fund managers. The 25 most successful fundraisers collected 41 percent of aggregate commitments to closed-end funds (with the top five managers accounting for nearly half that total). Closed-end fundraising totals may understate the extent of concentration in the industry overall, as the largest managers also tend to be more successful in raising non-institutional capital.

While the largest funds grew even larger—the largest vehicles on record were raised in buyout, real estate, infrastructure, and private debt in 2023—smaller and newer funds struggled. Fewer than 1,700 funds of less than $1 billion were closed during the year, half as many as closed in 2022 and the fewest of any year since 2012. New manager formation also fell to the lowest level since 2012, with just 651 new firms launched in 2023.

Whether recent fundraising concentration and a spate of M&A activity signals the beginning of oft-rumored consolidation in the private markets remains uncertain, as a similar pattern developed in each of the last two fundraising downturns before giving way to renewed entrepreneurialism among general partners (GPs) and commitment diversification among LPs. Compared with how things played out in the last two downturns, perhaps this movie really is different, or perhaps we’re watching a trilogy reusing a familiar plotline.

Dry powder inventory spiked (again)

Private markets assets under management totaled $13.1 trillion as of June 30, 2023, and have grown nearly 20 percent per annum since 2018. Dry powder reserves—the amount of capital committed but not yet deployed—increased to $3.7 trillion, marking the ninth consecutive year of growth. Dry powder inventory—the amount of capital available to GPs expressed as a multiple of annual deployment—increased for the second consecutive year in PE, as new commitments continued to outpace deal activity. Inventory sat at 1.6 years in 2023, up markedly from the 0.9 years recorded at the end of 2021 but still within the historical range. NAV grew as well, largely driven by the reluctance of managers to exit positions and crystallize returns in a depressed multiple environment.

Private equity strategies diverged

Buyout and venture capital, the two largest PE sub-asset classes, charted wildly different courses over the past 18 months. Buyout notched its highest fundraising year ever in 2023, and its performance improved, with funds posting a (still paltry) 5 percent net internal rate of return through September 30. And although buyout deal volumes declined by 19 percent, 2023 was still the third-most-active year on record. In contrast, venture capital (VC) fundraising declined by nearly 60 percent, equaling its lowest total since 2015, and deal volume fell by 36 percent to the lowest level since 2019. VC funds returned –3 percent through September, posting negative returns for seven consecutive quarters. VC was the fastest-growing—as well as the highest-performing—PE strategy by a significant margin from 2010 to 2022, but investors appear to be reevaluating their approach in the current environment.

Private equity entry multiples contracted

PE buyout entry multiples declined by roughly one turn from 11.9 to 11.0 times EBITDA, slightly outpacing the decline in public market multiples (down from 12.1 to 11.3 times EBITDA), through the first nine months of 2023. For nearly a decade leading up to 2022, managers consistently sold assets into a higher-multiple environment than that in which they had bought those assets, providing a substantial performance tailwind for the industry. Nowhere has this been truer than in technology. After experiencing more than eight turns of multiple expansion from 2009 to 2021 (the most of any sector), technology multiples have declined by nearly three turns in the past two years, 50 percent more than in any other sector. Overall, roughly two-thirds of the total return for buyout deals that were entered in 2010 or later and exited in 2021 or before can be attributed to market multiple expansion and leverage. Now, with falling multiples and higher financing costs, revenue growth and margin expansion are taking center stage for GPs.

Real estate receded

Demand uncertainty, slowing rent growth, and elevated financing costs drove cap rates higher and made price discovery challenging, all of which weighed on deal volume, fundraising, and investment performance. Global closed-end fundraising declined 34 percent year over year, and funds returned −4 percent in the first nine months of the year, losing money for the first time since the 2007–08 global financial crisis. Capital shifted away from core and core-plus strategies as investors sought liquidity via redemptions in open-end vehicles, from which net outflows reached their highest level in at least two decades. Opportunistic strategies benefited from this shift, with investors focusing on capital appreciation over income generation in a market where alternative sources of yield have grown more attractive. Rising interest rates widened bid–ask spreads and impaired deal volume across food groups, including in what were formerly hot sectors: multifamily and industrial.

Private debt pays dividends

Debt again proved to be the most resilient private asset class against a turbulent market backdrop. Fundraising declined just 13 percent, largely driven by lower commitments to direct lending strategies, for which a slower PE deal environment has made capital deployment challenging. The asset class also posted the highest returns among all private asset classes through September 30. Many private debt securities are tied to floating rates, which enhance returns in a rising-rate environment. Thus far, managers appear to have successfully navigated the rising incidence of default and distress exhibited across the broader leveraged-lending market. Although direct lending deal volume declined from 2022, private lenders financed an all-time high 59 percent of leveraged buyout transactions last year and are now expanding into additional strategies to drive the next era of growth.

Infrastructure took a detour

After several years of robust growth and strong performance, infrastructure and natural resources fundraising declined by 53 percent to the lowest total since 2013. Supply-side timing is partially to blame: five of the seven largest infrastructure managers closed a flagship vehicle in 2021 or 2022, and none of those five held a final close last year. As in real estate, investors shied away from core and core-plus investments in a higher-yield environment. Yet there are reasons to believe infrastructure’s growth will bounce back. Limited partners (LPs) surveyed by McKinsey remain bullish on their deployment to the asset class, and at least a dozen vehicles targeting more than $10 billion were actively fundraising as of the end of 2023. Multiple recent acquisitions of large infrastructure GPs by global multi-asset-class managers also indicate marketwide conviction in the asset class’s potential.

Private markets still have work to do on diversity

Private markets firms are slowly improving their representation of females (up two percentage points over the prior year) and ethnic and racial minorities (up one percentage point). On some diversity metrics, including entry-level representation of women, private markets now compare favorably with corporate America. Yet broad-based parity remains elusive and too slow in the making. Ethnic, racial, and gender imbalances are particularly stark across more influential investing roles and senior positions. In fact, McKinsey’s research  reveals that at the current pace, it would take several decades for private markets firms to reach gender parity at senior levels. Increasing representation across all levels will require managers to take fresh approaches to hiring, retention, and promotion.

Artificial intelligence generating excitement

The transformative potential of generative AI was perhaps 2023’s hottest topic (beyond Taylor Swift). Private markets players are excited about the potential for the technology to optimize their approach to thesis generation, deal sourcing, investment due diligence, and portfolio performance, among other areas. While the technology is still nascent and few GPs can boast scaled implementations, pilot programs are already in flight across the industry, particularly within portfolio companies. Adoption seems nearly certain to accelerate throughout 2024.

Private markets in a slower era

If private markets investors entered 2023 hoping for a return to the heady days of 2021, they likely left the year disappointed. Many of the headwinds that emerged in the latter half of 2022 persisted throughout the year, pressuring fundraising, dealmaking, and performance. Inflation moderated somewhat over the course of the year but remained stubbornly elevated by recent historical standards. Interest rates started high and rose higher, increasing the cost of financing. A reinvigorated public equity market recovered most of 2022’s losses but did little to resolve the valuation uncertainty private market investors have faced for the past 18 months.

Within private markets, the denominator effect remained in play, despite the public market recovery, as the numerator continued to expand. An activity-dampening cycle emerged: higher cost of capital and lower multiples limited the ability or willingness of general partners (GPs) to exit positions; fewer exits, coupled with continuing capital calls, pushed LP allocations higher, thereby limiting their ability or willingness to make new commitments. These conditions weighed on managers’ ability to fundraise. Based on data reported as of year-end 2023, private markets fundraising fell 22 percent from the prior year to just over $1 trillion, the largest such drop since 2009 (Exhibit 1).

The impact of the fundraising environment was not felt equally among GPs. Continuing a trend that emerged in 2022, and consistent with prior downturns in fundraising, LPs favored larger vehicles and the scaled GPs that typically manage them. Smaller and newer managers struggled, and the number of sub–$1 billion vehicles and new firm launches each declined to its lowest level in more than a decade.

Despite the decline in fundraising, private markets assets under management (AUM) continued to grow, increasing 12 percent to $13.1 trillion as of June 30, 2023. 2023 fundraising was still the sixth-highest annual haul on record, pushing dry powder higher, while the slowdown in deal making limited distributions.

Investment performance across private market asset classes fell short of historical averages. Private equity (PE) got back in the black but generated the lowest annual performance in the past 15 years, excluding 2022. Closed-end real estate produced negative returns for the first time since 2009, as capitalization (cap) rates expanded across sectors and rent growth dissipated in formerly hot sectors, including multifamily and industrial. The performance of infrastructure funds was less than half of its long-term average and even further below the double-digit returns generated in 2021 and 2022. Private debt was the standout performer (if there was one), outperforming all other private asset classes and illustrating the asset class’s countercyclical appeal.

Private equity down but not out

Higher financing costs, lower multiples, and an uncertain macroeconomic environment created a challenging backdrop for private equity managers in 2023. Fundraising declined for the second year in a row, falling 15 percent to $649 billion, as LPs grappled with the denominator effect and a slowdown in distributions. Managers were on the fundraising trail longer to raise this capital: funds that closed in 2023 were open for a record-high average of 20.1 months, notably longer than 18.7 months in 2022 and 14.1 months in 2018. VC and growth equity strategies led the decline, dropping to their lowest level of cumulative capital raised since 2015. Fundraising in Asia fell for the fourth year of the last five, with the greatest decline in China.

Despite the difficult fundraising context, a subset of strategies and managers prevailed. Buyout managers collectively had their best fundraising year on record, raising more than $400 billion. Fundraising in Europe surged by more than 50 percent, resulting in the region’s biggest haul ever. The largest managers raised an outsized share of the total for a second consecutive year, making 2023 the most concentrated fundraising year of the last decade (Exhibit 2).

Despite the drop in aggregate fundraising, PE assets under management increased 8 percent to $8.2 trillion. Only a small part of this growth was performance driven: PE funds produced a net IRR of just 2.5 percent through September 30, 2023. Buyouts and growth equity generated positive returns, while VC lost money. PE performance, dating back to the beginning of 2022, remains negative, highlighting the difficulty of generating attractive investment returns in a higher interest rate and lower multiple environment. As PE managers devise value creation strategies to improve performance, their focus includes ensuring operating efficiency and profitability of their portfolio companies.

Deal activity volume and count fell sharply, by 21 percent and 24 percent, respectively, which continued the slower pace set in the second half of 2022. Sponsors largely opted to hold assets longer rather than lock in underwhelming returns. While higher financing costs and valuation mismatches weighed on overall deal activity, certain types of M&A gained share. Add-on deals, for example, accounted for a record 46 percent of total buyout deal volume last year.

Real estate recedes

For real estate, 2023 was a year of transition, characterized by a litany of new and familiar challenges. Pandemic-driven demand issues continued, while elevated financing costs, expanding cap rates, and valuation uncertainty weighed on commercial real estate deal volumes, fundraising, and investment performance.

Managers faced one of the toughest fundraising environments in many years. Global closed-end fundraising declined 34 percent to $125 billion. While fundraising challenges were widespread, they were not ubiquitous across strategies. Dollars continued to shift to large, multi-asset class platforms, with the top five managers accounting for 37 percent of aggregate closed-end real estate fundraising. In April, the largest real estate fund ever raised closed on a record $30 billion.

Capital shifted away from core and core-plus strategies as investors sought liquidity through redemptions in open-end vehicles and reduced gross contributions to the lowest level since 2009. Opportunistic strategies benefited from this shift, as investors turned their attention toward capital appreciation over income generation in a market where alternative sources of yield have grown more attractive.

In the United States, for instance, open-end funds, as represented by the National Council of Real Estate Investment Fiduciaries Fund Index—Open-End Equity (NFI-OE), recorded $13 billion in net outflows in 2023, reversing the trend of positive net inflows throughout the 2010s. The negative flows mainly reflected $9 billion in core outflows, with core-plus funds accounting for the remaining outflows, which reversed a 20-year run of net inflows.

As a result, the NAV in US open-end funds fell roughly 16 percent year over year. Meanwhile, global assets under management in closed-end funds reached a new peak of $1.7 trillion as of June 2023, growing 14 percent between June 2022 and June 2023.

Real estate underperformed historical averages in 2023, as previously high-performing multifamily and industrial sectors joined office in producing negative returns caused by slowing demand growth and cap rate expansion. Closed-end funds generated a pooled net IRR of −3.5 percent in the first nine months of 2023, losing money for the first time since the global financial crisis. The lone bright spot among major sectors was hospitality, which—thanks to a rush of postpandemic travel—returned 10.3 percent in 2023. 2 Based on NCREIFs NPI index. Hotels represent 1 percent of total properties in the index. As a whole, the average pooled lifetime net IRRs for closed-end real estate funds from 2011–20 vintages remained around historical levels (9.8 percent).

Global deal volume declined 47 percent in 2023 to reach a ten-year low of $650 billion, driven by widening bid–ask spreads amid valuation uncertainty and higher costs of financing (Exhibit 3). 3 CBRE, Real Capital Analytics Deal flow in the office sector remained depressed, partly as a result of continued uncertainty in the demand for space in a hybrid working world.

During a turbulent year for private markets, private debt was a relative bright spot, topping private markets asset classes in terms of fundraising growth, AUM growth, and performance.

Fundraising for private debt declined just 13 percent year over year, nearly ten percentage points less than the private markets overall. Despite the decline in fundraising, AUM surged 27 percent to $1.7 trillion. And private debt posted the highest investment returns of any private asset class through the first three quarters of 2023.

Private debt’s risk/return characteristics are well suited to the current environment. With interest rates at their highest in more than a decade, current yields in the asset class have grown more attractive on both an absolute and relative basis, particularly if higher rates sustain and put downward pressure on equity returns (Exhibit 4). The built-in security derived from debt’s privileged position in the capital structure, moreover, appeals to investors that are wary of market volatility and valuation uncertainty.

Direct lending continued to be the largest strategy in 2023, with fundraising for the mostly-senior-debt strategy accounting for almost half of the asset class’s total haul (despite declining from the previous year). Separately, mezzanine debt fundraising hit a new high, thanks to the closings of three of the largest funds ever raised in the strategy.

Over the longer term, growth in private debt has largely been driven by institutional investors rotating out of traditional fixed income in favor of private alternatives. Despite this growth in commitments, LPs remain underweight in this asset class relative to their targets. In fact, the allocation gap has only grown wider in recent years, a sharp contrast to other private asset classes, for which LPs’ current allocations exceed their targets on average. According to data from CEM Benchmarking, the private debt allocation gap now stands at 1.4 percent, which means that, in aggregate, investors must commit hundreds of billions in net new capital to the asset class just to reach current targets.

Private debt was not completely immune to the macroeconomic conditions last year, however. Fundraising declined for the second consecutive year and now sits 23 percent below 2021’s peak. Furthermore, though private lenders took share in 2023 from other capital sources, overall deal volumes also declined for the second year in a row. The drop was largely driven by a less active PE deal environment: private debt is predominantly used to finance PE-backed companies, though managers are increasingly diversifying their origination capabilities to include a broad new range of companies and asset types.

Infrastructure and natural resources take a detour

For infrastructure and natural resources fundraising, 2023 was an exceptionally challenging year. Aggregate capital raised declined 53 percent year over year to $82 billion, the lowest annual total since 2013. The size of the drop is particularly surprising in light of infrastructure’s recent momentum. The asset class had set fundraising records in four of the previous five years, and infrastructure is often considered an attractive investment in uncertain markets.

While there is little doubt that the broader fundraising headwinds discussed elsewhere in this report affected infrastructure and natural resources fundraising last year, dynamics specific to the asset class were at play as well. One issue was supply-side timing: nine of the ten largest infrastructure GPs did not close a flagship fund in 2023. Second was the migration of investor dollars away from core and core-plus investments, which have historically accounted for the bulk of infrastructure fundraising, in a higher rate environment.

The asset class had some notable bright spots last year. Fundraising for higher-returning opportunistic strategies more than doubled the prior year’s total (Exhibit 5). AUM grew 18 percent, reaching a new high of $1.5 trillion. Infrastructure funds returned a net IRR of 3.4 percent in 2023; this was below historical averages but still the second-best return among private asset classes. And as was the case in other asset classes, investors concentrated commitments in larger funds and managers in 2023, including in the largest infrastructure fund ever raised.

The outlook for the asset class, moreover, remains positive. Funds targeting a record amount of capital were in the market at year-end, providing a robust foundation for fundraising in 2024 and 2025. A recent spate of infrastructure GP acquisitions signal multi-asset managers’ long-term conviction in the asset class, despite short-term headwinds. Global megatrends like decarbonization and digitization, as well as revolutions in energy and mobility, have spurred new infrastructure investment opportunities around the world, particularly for value-oriented investors that are willing to take on more risk.

Private markets make measured progress in DEI

Diversity, equity, and inclusion (DEI) has become an important part of the fundraising, talent, and investing landscape for private market participants. Encouragingly, incremental progress has been made in recent years, including more diverse talent being brought to entry-level positions, investing roles, and investment committees. The scope of DEI metrics provided to institutional investors during fundraising has also increased in recent years: more than half of PE firms now provide data across investing teams, portfolio company boards, and portfolio company management (versus investment team data only). 4 “ The state of diversity in global private markets: 2023 ,” McKinsey, August 22, 2023.

In 2023, McKinsey surveyed 66 global private markets firms that collectively employ more than 60,000 people for the second annual State of diversity in global private markets report. 5 “ The state of diversity in global private markets: 2023 ,” McKinsey, August 22, 2023. The research offers insight into the representation of women and ethnic and racial minorities in private investing as of year-end 2022. In this chapter, we discuss where the numbers stand and how firms can bring a more diverse set of perspectives to the table.

The statistics indicate signs of modest advancement. Overall representation of women in private markets increased two percentage points to 35 percent, and ethnic and racial minorities increased one percentage point to 30 percent (Exhibit 6). Entry-level positions have nearly reached gender parity, with female representation at 48 percent. The share of women holding C-suite roles globally increased 3 percentage points, while the share of people from ethnic and racial minorities in investment committees increased 9 percentage points. There is growing evidence that external hiring is gradually helping close the diversity gap, especially at senior levels. For example, 33 percent of external hires at the managing director level were ethnic or racial minorities, higher than their existing representation level (19 percent).

Yet, the scope of the challenge remains substantial. Women and minorities continue to be underrepresented in senior positions and investing roles. They also experience uneven rates of progress due to lower promotion and higher attrition rates, particularly at smaller firms. Firms are also navigating an increasingly polarized workplace today, with additional scrutiny and a growing number of lawsuits against corporate diversity and inclusion programs, particularly in the US, which threatens to impact the industry’s pace of progress.

Fredrik Dahlqvist is a senior partner in McKinsey’s Stockholm office; Alastair Green  is a senior partner in the Washington, DC, office, where Paul Maia and Alexandra Nee  are partners; David Quigley  is a senior partner in the New York office, where Connor Mangan is an associate partner and Aditya Sanghvi  is a senior partner; Rahel Schneider is an associate partner in the Bay Area office; John Spivey is a partner in the Charlotte office; and Brian Vickery  is a partner in the Boston office.

The authors wish to thank Jonathan Christy, Louis Dufau, Vaibhav Gujral, Graham Healy-Day, Laura Johnson, Ryan Luby, Tripp Norton, Alastair Rami, Henri Torbey, and Alex Wolkomir for their contributions

The authors would also like to thank CEM Benchmarking and the StepStone Group for their partnership in this year's report.

This article was edited by Arshiya Khullar, an editor in the Gurugram office.

Explore a career with us

Related articles.

CEO alpha: A new approach to generating private equity outperformance

Private equity turns to resiliency strategies for software investments

The state of diversity in global private markets: 2022