case study assessment

• Numerical Reasoning
• Verbal Reasoning
• Inductive Reasoning
• Logical Reasoning
• Situational Judgement
• Mechanical Reasoning
• Watson Glaser Critical thinking
• Deductive reasoning
• Abstract reasoning
• Spatial reasoning
• Error checking
• Verbal comprehension
• Reading comprehension
• Diagrammatic Reasoning
• Psychometric tests
• Personality test
• In-Tray exercise
• E-Tray exercise
• Competency based assessment
• Game based assessments
• Analysis exercise
• Group exercise
• Presentation exercise
• Video interview
• Strengths based assessment
• Strengths based interviews
• Saville Assessment
• Talent Q / Korn Ferry
• Watson Glaser
• Criterion Partnership
• Test Partnership
• Cut-e / Aon
• Team Focus PFS
• Sova Assessment

Chapter 6: Case Study Exercises

A resource guide to help you master case study exercises

Page contents:

What is a case study exercise, how to answer a case study exercise, what skills does a case-study exercise assess, what questions will be asked in a case study exercise, case study exercise tips to succeed, key takeaways.

Case-study exercises are a very popular part of an assessment centre. But don't worry, with a bit of preparation and understanding, you can ace this part of the assessment.

Case study exercises are a popular tool used by employers to evaluate candidates' problem-solving skills, analytical thinking, and decision-making abilities. These exercises can be in the form of a written report, a presentation, or a group discussion, and typically involve a hypothetical business problem that requires a solution.

The case study presents the candidate with a series of fictional documents such as company reports, a consultant’s report, results from new product research etc. (i.e. similar to the in-tray exercise except these documents will be longer). You will then be asked to make business decisions based on the information. This can be done as an individual exercise, or more likely done in a group discussion so that assessors can also score your teamworking ability.

Before you start the exercise, it's important to carefully read and understand the instructions. Make sure you know what you're being asked to do, what resources you have available to you, and how your performance will be assessed. If you're unsure about anything, don't be afraid to ask for clarification.

Once you've read the case study, it's time to start analysing the problem. This involves breaking down the problem into its component parts, identifying the key issues, and considering different options for addressing them. It's important to approach the problem from different angles and to consider the implications of each possible solution.

During the exercise, you'll need to demonstrate your ability to work well under pressure, to think on your feet, and to communicate your ideas effectively. Make sure to use clear and concise language, and to back up your arguments with evidence and examples.

If you're working on a group case study exercise, it's important to listen to the ideas of others and to contribute your own ideas in a constructive and respectful way. Remember that the assessors are not only evaluating your individual performance but also how well you work as part of a team.

When it comes to presenting your solution, make sure to structure your presentation in a clear and logical way. Start with an introduction that sets out the problem and your approach, then move onto your analysis and recommendations, and finish with a conclusion that summarizes your key points. Make sure to keep to time and to engage your audience with your presentation.

A case study exercise is designed to assess several core competencies that are critical for success in the role you are applying for. There will be many common competencies that will be valuable across most roles in the professional world, these competencies typically include:

• Problem-Solving Skills: The ability to identify and analyse problems, and to develop and implement effective solutions.
• Analytical Thinking: The capacity to break down complex information into smaller parts, evaluate it systematically, and draw meaningful conclusions.
• Decision-Making Abilities: The ability to make well-informed and timely decisions, considering all relevant information and potential outcomes.
• Communication Skills: The capacity to convey ideas clearly and concisely, and to listen actively to others.
• Teamwork Skills: The ability to collaborate effectively with others, and to work towards a shared goal.
• Time Management: The capacity to prioritise tasks and to manage time effectively, while maintaining quality and meeting deadlines.

By assessing these competencies, employers can gain valuable insights into how candidates approach problems, how they think critically, and how they work with others to achieve goals. Ultimately, the aim is to identify candidates who can add value to the organisation, and who have the potential to become successful and productive members of the team.

Different companies will prioritise certain competencies; the original job description is a great place to look for finding out what competencies the employer desires and so will likely be scoring you against during the assessment centre activities.

The type of questions that may be asked can vary, but here are some examples of the most common types:

• Analytical Questions: These questions require the candidate to analyse a set of data or information and draw conclusions based on their findings. For example: "You have been given a dataset on customer behaviour. What insights can you draw from the data to improve sales performance?"
• Decision-Making Questions: These questions ask the candidate to make a decision based on a given scenario. For example: "You are the CEO of a company that is considering a merger. What factors would you consider when making the decision to proceed with the merger?"
• Group Discussion Questions: In a group case study exercise, candidates may be asked to work together to analyse a problem and present their findings to the assessors. For example: "As a team, analyse the strengths and weaknesses of our company's current marketing strategy and recommend improvements."

The questions are designed to test the candidate's problem-solving, analytical thinking, decision-making, and communication skills. It's important to carefully read and understand the questions, and to provide well-reasoned and evidence-based responses.

It has been known for employers to use real live projects for the case study exercise with sensitive information swapped for fictional examples.

Information from the case study exercise lends itself to be used as scene-setting for other exercises at the assessment centre. It is common to have the same fictional setting running through the assessment centre, to save time on having to describe a new scenario for each task. You will be told in each exercise if you are expected to remember the information from a previous exercise, but this is rarely the case. Usually the only information common to multiple exercises is the fictional scenario; all data to be used in each exercise will be part of that exercise.

Don't leave your future career down to chance!

Get a completely free starter account

• Over 20 tests
• Track your progress

Here are some key tips to help you prepare for and successfully pass a case study exercise at an assessment centre:

• Understand the Brief: Carefully read and analyse the case study brief, making sure you understand the problem or scenario being presented, and the information and data provided. Take notes and identify key issues and opportunities.
• Plan Your Approach: Take some time to plan your approach to the case study exercise. Consider the key challenges and opportunities, and identify potential solutions and recommendations. This will help you structure your thoughts and prioritise your ideas.
• Use Evidence: Use evidence from the case study, as well as your own research and knowledge, to support your ideas and recommendations. This will demonstrate your analytical thinking and problem-solving skills.
• Stay Focused: During the exercise, stay focused on the task at hand and avoid getting sidetracked by irrelevant information or details. Keep the objective of the exercise in mind, and stay on track with your analysis and recommendations.
• Collaborate Effectively: If the case study exercise involves group work, make sure to communicate clearly and effectively with your team members. Listen actively to their ideas, and contribute constructively to the discussion.
• Be Confident: Have confidence in your ideas and recommendations, and be prepared to defend your positions if challenged. Speak clearly and confidently, and use evidence and data to support your arguments.

Here is the summary of what case-study exercises are and how to pass them:

• A case study exercise is a type of assessment where candidates are presented with a hypothetical business scenario and asked to provide solutions or recommendations.
• These exercises assess a range of competencies such as problem-solving, analytical thinking, decision-making, communication, teamwork, and time management.
• To pass a case study exercise, it's important to carefully read and understand the brief, plan your approach, use evidence to support your ideas, stay focused, collaborate effectively, be confident, and manage your time effectively.

Fully understanding the format of the exercise, taking practice case-study exercises and following our tips outlined above will drastically improve the chances of you standing out as a star candidate at the assessment centre.

Assessment by Case Studies and Scenarios

Case studies depict real-life situations in which problems need to be solved. Scenario-based teaching may be similar to case studies, or may be oriented toward developing communication or teamwork skills. Both case studies and scenarios are commonly used methods of problem-based learning. Typically, using these methods, teachers aim to develop student reasoning, problem-solving and decision-making skills. Case studies differ from role plays in that in the former, learning takes place largely through discussion and analysis, whereas in the latter, students assume a character or role and "act out" what that character would do in the scenario (The Teaching Gateway page Assessing with Role Plays and Simulations contains more information on using role plays for assessments.) Like role plays and simulations, case studies and scenarios aim for authenticity:  allowing students to get a sense of the situations they might face in the real world upon graduation. Students can see how their learning and skills can be applied in a real-world situation, without the pressure of being actually involved in that situation with the associated constraints on research, discussion and reflection time.

Case studies and scenarios are particularly useful when they present situations are complex and solutions are uncertain. Ideally, their complexity requires group members to draw from and share their experiences, work together, and learn by doing to understand and help solve the case-study problem.

You can present a single case to several groups in a class and require each group to offer its solutions, or you can give a different case to each group or individual.

Case studies' effectiveness comes from their abiliity to:

• engage students in research and reflective discussion
• encourage clinical and professional reasoning in a safe environment
• encourage higher-order thinking
• facilitate creative problem solving and the application of different problem-solving theories without risk to third parties or projects
• allow students to develop realistic solutions to complex problems
• develop students' ability to identify and distinguish between critical and extraneous factors
• enable students to apply previously acquired skills
• allow students to learn from one another
• provide an effective simulated learning environment
• encourage practical reasoning
• allow you to assess individuals or teams.

You can use case studies to bridge the gap between teacher-centred lectures and pure problem-based learning. They leave room for you to guide students directly, while the scenarios themselves suggest how students should operate, and provide parameters for their work.

Although some students have reported greater satisfaction with simulations as learning tools than with case studies (Maamari & El-Nakla, 2023), case studies generally require less up-front preparation time, and can be less intimidating for students.

To make case studies an effective form of assessment, instructors and tutors need to be familiar with their use in both teaching and assessment. This applies whether teachers are developing the case studies for their courses themselves or using those developed by others.

Case studies reach their highest effectiveness as a teaching and assessment tool when they are authentic; ensuring that case studies accurately reflect the circumstances in which a student will eventually be practising professionally can require a considerable amount of research, as well as the potential involvement of industry professionals.

Students may need scaffolding as they learn how to problem-solve in the context of case studies; using case studies as low-stakes, formative assessments can prepare them for summative assessment by case study at the end of the course.

Learning outcomes, course outlines, and marking rubrics need to be entirely clear about how case studies will be used in the course and how students will be expected to demonstrate their learning through thee way they analyse and problem-solve in the context of case studies.

Assessment preparation

Typically, the product assessed after case study or scenario work is a verbal presentation or a written submission. Decide who will take part in the assessment: the tutor, an industry specialist, a panel, peer groups or students themselves by self-evaluation? Choose whether to give a class or group mark or to assess individual performance; and whether to assess the product yourself or have it assessed by peers.

Assessment strategies

You can assess students’ interaction with other members of a group by asking open-ended questions, and setting tasks that require teamwork and sharing resources.

Assess the process of analysis

Case studies allow you to assess a student’s demonstration of deeper understanding and cognitive skills as they address the case.  These skills include, for example:

• identification of a problem
• hypotheses generation
• construction of an enquiry plan
• interpretation of findings
• investigation of results collected for evidence to refine a hypothesis and construction of a management plan.

During the problem-solving process, you can also observe and evaluate:

• quality of research
• structural issues in written material
• organisation of arguments
• feasibility of solutions presented
• intra-group dynamics
• evidence of consideration of all case factors
• multiple resolutions of the same scenario issue.

Use a variety of questions in case analysis

The Questioning page discusses in detail various ways to use questions in teaching . If your students are using the Harvard Business School case study method for their analysis, use a range of question types to enable the class to move through the stages of analysis:

• clarification/information seeking ( What? )
• analysis/diagnosis ( Why? )
• conclusion/recommendation ( What now? )
• implementation ( How? ) and
• application/reflection ( So what? What does it mean to you?)

Use technology

Learning-management systems such as Moodle can help you track contributions to case discussions . You can assess students' interactions with other members of a group by viewing their responses to open-ended questions or observing their teamwork and sharing of resources as part of the discussion.  You can incorporate the use of various tools in these systems, or others such as Survey Monkey, into students' assessment of their peers, or of their group members' contribution to exploring and presenting case studies. You can also set this peer assessment up so that it takes place anonymously.

Assessing by Case Studies: UNSW examples

These videos show examples of how UNSW faculty have implemented case studies in their own courses.

• Boston University. Using Case Studies to Teach
• Columbia University. Case Method Teaching and Learning
• Science Education Resource Center, Carleton College. Starting Point: What is Investigative Case-Based Learning?

Maamari, B. E., & El-Nakla, D. (2023). From case studies to experiential learning. Is simulation an effective tool for student assessment? Arab Economic and Business Journal, 15(1), Article 2. https://doi.org/10.38039/2214-4625.1023

Merret, C. (2020). Using case studies and build projects as authentic assessments in cornerstone courses. International Journal of Mechanical Engineering Education , 50 (1), 20-50. https://doi.org/10.1177/0306419020913286

Porzecanski, A. L., Bravo, A., Groom, M. J., Dávalos, L. M., Bynum, N., Abraham, B. J., Cigliano, J. A., Griffiths, C., Stokes, D. L., Cawthorn, M., Fernandez, D. S., Freeman,  L., Leslie, T., Theodose, T., Vogler, D., & Sterling, E. J. (2021). Using case studies to improve the critical thinking skills of undergraduate conservation biology students. Case Studies in the Environment , 5 (1), 1536396. https://doi.org/10.1525/cse.2021.1536396

• Teaching for learning
• Assessment Toolkit
• Digital Assessment at UNSW
• Designing Assessment
• Selecting Assessment Methods
• Selecting Assessment Technologies
• Oral Assessment
• Capstone Project
• Case Studies & Scenarios
• Oral Presentations
• Role Play & Simulation
• Grading & Giving Feedback
• Reviewing Assessment Quality
• Spotlight on Assessment
• Assessment Development Framework
• Teaching Settings

Events & news

• Bipolar Disorder
• Therapy Center
• When To See a Therapist
• Types of Therapy
• Best Online Therapy
• Best Couples Therapy
• Best Family Therapy
• Managing Stress
• Sleep and Dreaming
• Understanding Emotions
• Self-Improvement
• Healthy Relationships
• Student Resources
• Personality Types
• Guided Meditations
• Verywell Mind Insights
• 2024 Verywell Mind 25
• Mental Health in the Classroom
• Editorial Process
• Meet Our Review Board
• Crisis Support

What Is a Case Study?

Weighing the pros and cons of this method of research

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Cara Lustik is a fact-checker and copywriter.

Verywell / Colleen Tighe

• Pros and Cons

What Types of Case Studies Are Out There?

Where do you find data for a case study, how do i write a psychology case study.

A case study is an in-depth study of one person, group, or event. In a case study, nearly every aspect of the subject's life and history is analyzed to seek patterns and causes of behavior. Case studies can be used in many different fields, including psychology, medicine, education, anthropology, political science, and social work.

The point of a case study is to learn as much as possible about an individual or group so that the information can be generalized to many others. Unfortunately, case studies tend to be highly subjective, and it is sometimes difficult to generalize results to a larger population.

While case studies focus on a single individual or group, they follow a format similar to other types of psychology writing. If you are writing a case study, we got you—here are some rules of APA format to reference.  

At a Glance

A case study, or an in-depth study of a person, group, or event, can be a useful research tool when used wisely. In many cases, case studies are best used in situations where it would be difficult or impossible for you to conduct an experiment. They are helpful for looking at unique situations and allow researchers to gather a lot of˜ information about a specific individual or group of people. However, it's important to be cautious of any bias we draw from them as they are highly subjective.

What Are the Benefits and Limitations of Case Studies?

A case study can have its strengths and weaknesses. Researchers must consider these pros and cons before deciding if this type of study is appropriate for their needs.

One of the greatest advantages of a case study is that it allows researchers to investigate things that are often difficult or impossible to replicate in a lab. Some other benefits of a case study:

• Allows researchers to capture information on the 'how,' 'what,' and 'why,' of something that's implemented
• Gives researchers the chance to collect information on why one strategy might be chosen over another
• Permits researchers to develop hypotheses that can be explored in experimental research

On the other hand, a case study can have some drawbacks:

• It cannot necessarily be generalized to the larger population
• Cannot demonstrate cause and effect
• It may not be scientifically rigorous
• It can lead to bias

Researchers may choose to perform a case study if they want to explore a unique or recently discovered phenomenon. Through their insights, researchers develop additional ideas and study questions that might be explored in future studies.

It's important to remember that the insights from case studies cannot be used to determine cause-and-effect relationships between variables. However, case studies may be used to develop hypotheses that can then be addressed in experimental research.

Case Study Examples

There have been a number of notable case studies in the history of psychology. Much of  Freud's work and theories were developed through individual case studies. Some great examples of case studies in psychology include:

• Anna O : Anna O. was a pseudonym of a woman named Bertha Pappenheim, a patient of a physician named Josef Breuer. While she was never a patient of Freud's, Freud and Breuer discussed her case extensively. The woman was experiencing symptoms of a condition that was then known as hysteria and found that talking about her problems helped relieve her symptoms. Her case played an important part in the development of talk therapy as an approach to mental health treatment.
• Phineas Gage : Phineas Gage was a railroad employee who experienced a terrible accident in which an explosion sent a metal rod through his skull, damaging important portions of his brain. Gage recovered from his accident but was left with serious changes in both personality and behavior.
• Genie : Genie was a young girl subjected to horrific abuse and isolation. The case study of Genie allowed researchers to study whether language learning was possible, even after missing critical periods for language development. Her case also served as an example of how scientific research may interfere with treatment and lead to further abuse of vulnerable individuals.

Such cases demonstrate how case research can be used to study things that researchers could not replicate in experimental settings. In Genie's case, her horrific abuse denied her the opportunity to learn a language at critical points in her development.

This is clearly not something researchers could ethically replicate, but conducting a case study on Genie allowed researchers to study phenomena that are otherwise impossible to reproduce.

There are a few different types of case studies that psychologists and other researchers might use:

• Collective case studies : These involve studying a group of individuals. Researchers might study a group of people in a certain setting or look at an entire community. For example, psychologists might explore how access to resources in a community has affected the collective mental well-being of those who live there.
• Descriptive case studies : These involve starting with a descriptive theory. The subjects are then observed, and the information gathered is compared to the pre-existing theory.
• Explanatory case studies : These   are often used to do causal investigations. In other words, researchers are interested in looking at factors that may have caused certain things to occur.
• Exploratory case studies : These are sometimes used as a prelude to further, more in-depth research. This allows researchers to gather more information before developing their research questions and hypotheses .
• Instrumental case studies : These occur when the individual or group allows researchers to understand more than what is initially obvious to observers.
• Intrinsic case studies : This type of case study is when the researcher has a personal interest in the case. Jean Piaget's observations of his own children are good examples of how an intrinsic case study can contribute to the development of a psychological theory.

The three main case study types often used are intrinsic, instrumental, and collective. Intrinsic case studies are useful for learning about unique cases. Instrumental case studies help look at an individual to learn more about a broader issue. A collective case study can be useful for looking at several cases simultaneously.

The type of case study that psychology researchers use depends on the unique characteristics of the situation and the case itself.

There are a number of different sources and methods that researchers can use to gather information about an individual or group. Six major sources that have been identified by researchers are:

• Archival records : Census records, survey records, and name lists are examples of archival records.
• Direct observation : This strategy involves observing the subject, often in a natural setting . While an individual observer is sometimes used, it is more common to utilize a group of observers.
• Documents : Letters, newspaper articles, administrative records, etc., are the types of documents often used as sources.
• Interviews : Interviews are one of the most important methods for gathering information in case studies. An interview can involve structured survey questions or more open-ended questions.
• Participant observation : When the researcher serves as a participant in events and observes the actions and outcomes, it is called participant observation.
• Physical artifacts : Tools, objects, instruments, and other artifacts are often observed during a direct observation of the subject.

If you have been directed to write a case study for a psychology course, be sure to check with your instructor for any specific guidelines you need to follow. If you are writing your case study for a professional publication, check with the publisher for their specific guidelines for submitting a case study.

Here is a general outline of what should be included in a case study.

Section 1: A Case History

This section will have the following structure and content:

Background information : The first section of your paper will present your client's background. Include factors such as age, gender, work, health status, family mental health history, family and social relationships, drug and alcohol history, life difficulties, goals, and coping skills and weaknesses.

Description of the presenting problem : In the next section of your case study, you will describe the problem or symptoms that the client presented with.

Describe any physical, emotional, or sensory symptoms reported by the client. Thoughts, feelings, and perceptions related to the symptoms should also be noted. Any screening or diagnostic assessments that are used should also be described in detail and all scores reported.

Your diagnosis : Provide your diagnosis and give the appropriate Diagnostic and Statistical Manual code. Explain how you reached your diagnosis, how the client's symptoms fit the diagnostic criteria for the disorder(s), or any possible difficulties in reaching a diagnosis.

Section 2: Treatment Plan

This portion of the paper will address the chosen treatment for the condition. This might also include the theoretical basis for the chosen treatment or any other evidence that might exist to support why this approach was chosen.

• Cognitive behavioral approach : Explain how a cognitive behavioral therapist would approach treatment. Offer background information on cognitive behavioral therapy and describe the treatment sessions, client response, and outcome of this type of treatment. Make note of any difficulties or successes encountered by your client during treatment.
• Humanistic approach : Describe a humanistic approach that could be used to treat your client, such as client-centered therapy . Provide information on the type of treatment you chose, the client's reaction to the treatment, and the end result of this approach. Explain why the treatment was successful or unsuccessful.
• Psychoanalytic approach : Describe how a psychoanalytic therapist would view the client's problem. Provide some background on the psychoanalytic approach and cite relevant references. Explain how psychoanalytic therapy would be used to treat the client, how the client would respond to therapy, and the effectiveness of this treatment approach.
• Pharmacological approach : If treatment primarily involves the use of medications, explain which medications were used and why. Provide background on the effectiveness of these medications and how monotherapy may compare with an approach that combines medications with therapy or other treatments.

This section of a case study should also include information about the treatment goals, process, and outcomes.

When you are writing a case study, you should also include a section where you discuss the case study itself, including the strengths and limitiations of the study. You should note how the findings of your case study might support previous research. 

In your discussion section, you should also describe some of the implications of your case study. What ideas or findings might require further exploration? How might researchers go about exploring some of these questions in additional studies?

Need More Tips?

Here are a few additional pointers to keep in mind when formatting your case study:

• Never refer to the subject of your case study as "the client." Instead, use their name or a pseudonym.
• Read examples of case studies to gain an idea about the style and format.
• Remember to use APA format when citing references .

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach .  BMC Med Res Methodol . 2011;11:100.

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach . BMC Med Res Methodol . 2011 Jun 27;11:100. doi:10.1186/1471-2288-11-100

Gagnon, Yves-Chantal.  The Case Study as Research Method: A Practical Handbook . Canada, Chicago Review Press Incorporated DBA Independent Pub Group, 2010.

Yin, Robert K. Case Study Research and Applications: Design and Methods . United States, SAGE Publications, 2017.

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Get 25% off all test packages.

Get 25% off all test packages!

Click below to get 25% off all test packages.

Case Study Exercise At Assessment Centres

A case study exercise is a practical assessment commonly used in the latter stages of recruitment for graduate jobs. One of several activities undertaken at an assessment centre , this particular type of exercise allows employers to see your skills in action in a work-based context.

What is a case study exercise?

A case study exercise consists of a hypothetical scenario, similar to something you’d expect to encounter in daily working life. You’ll be tasked with examining information, drawing conclusions, and proposing business-based solutions for the situation at hand.

Information is typically presented in the form of fictional documentation: for example, market research findings, company reports, or details on a potential new venture. In some cases, it will be verbally communicated by the assessor.

You may also have additional or updated information drip-fed to you throughout the exercise.

You could be asked to work as an individual, but it’s more common to tackle a case study exercise as part of a group, since this shows a wider array of skills like teamwork and joint decision-making.

In both cases you’ll have a set amount of time to analyse the scenario and supporting information before presenting your findings, either through a written report or a presentation to an assessment panel. Here, you’ll need to explain your process and justify all decisions made.

Historically, assessment centres have been attended in person, but as more companies look to adopt virtual techniques, you may take part in a remote case study exercise. Depending on the employer and their platform of choice, this could be via pre-recorded content or a video conferencing tool that allows you to work alongside other candidates.

What competencies does a case study exercise assess?

There are multiple skills under assessment throughout a case study exercise. The most common are:

Problem solving

In itself, this involves various skills, like analytical thinking , creativity and innovation. How you approach your case study exercise will show employers how you’re likely to implement problem-solving skills in the work environment.

Show these at every stage of the process. If working in a group, be sure to make a contribution and be active in discussions, since assessors will be watching how you interact.

If working solo, explain your process to show problem solving in action.

Communication

How you present findings and communicate ideas is a major part of a case study exercise, as are other communication skills like effective listening.

Regardless of whether you present as an individual or a group, make sure you explain how you came to your conclusions, the evidence they’re based on and why you see them as effective.

Commercial awareness and business acumen

Assessors will be looking for a broader understanding of the industry in which the company operates and knowledge of best practice for growth.

Standout candidates will approach their case study with a business-first perspective, able to demonstrate how every decision made is rooted in organisational goals.

Decision making

At the heart of every case study exercise, there are key decisions to be made. Typically, there’s no right or wrong answer here, provided you can justify your decisions and back them up evidentially.

Along with problem solving, this is one of the top skills assessors are looking for, so don’t be hesitant. Make your decisions and stick to them.

Group exercises show assessors how well you work as part of a team, so make sure you’re actively involved, attentive and fair. Never dominate a discussion or press for your own agenda.

Approach all ideas equally and assess their pros and cons to arrive at the best solution.

What are the different types of case study exercise?

Depending on the role for which you’ve applied, you’ll either be presented with a general case study exercise or one related to a specific subject.

Subject-related case studies are used for roles where industry-specific knowledge is a prerequisite, and will be very much akin to the type of responsibilities you’ll be given if hired by the organisation.

For example, if applying for a role in mergers and acquisitions, you may be asked to assess the feasibility of a buy-out based on financial performance and market conditions.

General case studies are used to assess a wider pool of applicants for different positions. They do not require specific expertise, but rather rely on common sense and key competencies. All the information needed to complete the exercise will be made available to you.

Common topics covered in case study exercises include:

• The creation of new marketing campaigns
• Expansion through company or product acquisition
• Organisational change in terms of business structure
• Product or service diversification and entering new markets
• Strategic decision-making based on hypothetical influencing factors

Tips for performing well in case study exercises

1. process all the information.

Take time to fully understand the scenario and the objectives of the exercise, identify relevant information and highlight key points for analysis, or discussion if working as part of a team. This will help structure your approach in a logical manner.

2. Work collaboratively

In a group exercise , teamwork is vital. Assign roles based on individual skill sets. For example, if you’re a confident leader you may head up the exercise.

If you’re more of a listener, you may volunteer to keep notes. Avoid conflict by ensuring all points of view are heard and decisions made together.

3. Manage your time

Organisational skills and your ability to prioritise are both being evaluated, and since you have a set duration in which to complete the exercise, good time management is key.

Remember you also need to prepare a strong presentation, so allow plenty of scope for this.

Make an assertive decision

There’s no right answer to a case study exercise, but any conclusions you do draw should be evidenced-based and justifiable. Put forward solutions that you firmly believe in and can back up with solid reasoning.

5. Present your findings clearly

A case study exercise isn’t just about the decisions you make, but also how you articulate them. State your recommendations and then provide the background to your findings with clear, concise language and a confident presentation style.

If presenting as a group, assign specific sections to each person to avoid confusion.

How to prepare for a case study exercise

It’s unlikely you’ll know the nature of your case study exercise before your assessment day, but there are ways to prepare in advance. For a guide on the type of scenario you may face, review the job description or recruitment pack and look for key responsibilities.

You should also research the hiring organisation in full. Look into its company culture, read any recent press releases and refer to its social media to get a feel for both its day-to-day activities and wider achievements. Reading business news will also give you a good understanding of current issues relevant to the industry.

To improve your skills, carry out some practice case study exercises and present your findings to family or friends. This will get you used to the process and give you greater confidence on assessment centre day.

Choose a plan and start practising

Immediate access. Cancel anytime.

• 30 Numerical reasoning tests
• 30 Verbal reasoning tests
• 30 Diagrammatic reasoning tests
• 30 Situational judgement tests
• 34 Publisher packages e.g. Watson Glaser
• 252 Employer packages e.g. HSBC
• 29 Extra packages e.g Mechanical
• Dashboard performance tracking
• Full solutions and explanations
• Tips, tricks, guides and resources
• Access to free tests
• Basic performance tracking
• Solutions & explanations
• Tips and resources

By using our website you agree with our Cookie Policy.

Organizing Your Social Sciences Research Assignments

• Annotated Bibliography
• Analyzing a Scholarly Journal Article
• Group Presentations
• Dealing with Nervousness
• Using Visual Aids
• Grading Someone Else's Paper
• Types of Structured Group Activities
• Group Project Survival Skills
• Leading a Class Discussion
• Multiple Book Review Essay
• Reviewing Collected Works
• Writing a Case Analysis Paper
• Writing a Case Study
• About Informed Consent
• Writing Field Notes
• Writing a Policy Memo
• Writing a Reflective Paper
• Writing a Research Proposal
• Generative AI and Writing
• Acknowledgments

Definition and Introduction

Case analysis is a problem-based teaching and learning method that involves critically analyzing complex scenarios within an organizational setting for the purpose of placing the student in a “real world” situation and applying reflection and critical thinking skills to contemplate appropriate solutions, decisions, or recommended courses of action. It is considered a more effective teaching technique than in-class role playing or simulation activities. The analytical process is often guided by questions provided by the instructor that ask students to contemplate relationships between the facts and critical incidents described in the case.

Cases generally include both descriptive and statistical elements and rely on students applying abductive reasoning to develop and argue for preferred or best outcomes [i.e., case scenarios rarely have a single correct or perfect answer based on the evidence provided]. Rather than emphasizing theories or concepts, case analysis assignments emphasize building a bridge of relevancy between abstract thinking and practical application and, by so doing, teaches the value of both within a specific area of professional practice.

Given this, the purpose of a case analysis paper is to present a structured and logically organized format for analyzing the case situation. It can be assigned to students individually or as a small group assignment and it may include an in-class presentation component. Case analysis is predominately taught in economics and business-related courses, but it is also a method of teaching and learning found in other applied social sciences disciplines, such as, social work, public relations, education, journalism, and public administration.

Ellet, William. The Case Study Handbook: A Student's Guide . Revised Edition. Boston, MA: Harvard Business School Publishing, 2018; Christoph Rasche and Achim Seisreiner. Guidelines for Business Case Analysis . University of Potsdam; Writing a Case Analysis . Writing Center, Baruch College; Volpe, Guglielmo. "Case Teaching in Economics: History, Practice and Evidence." Cogent Economics and Finance 3 (December 2015). doi:https://doi.org/10.1080/23322039.2015.1120977.

How to Approach Writing a Case Analysis Paper

The organization and structure of a case analysis paper can vary depending on the organizational setting, the situation, and how your professor wants you to approach the assignment. Nevertheless, preparing to write a case analysis paper involves several important steps. As Hawes notes, a case analysis assignment “...is useful in developing the ability to get to the heart of a problem, analyze it thoroughly, and to indicate the appropriate solution as well as how it should be implemented” [p.48]. This statement encapsulates how you should approach preparing to write a case analysis paper.

Before you begin to write your paper, consider the following analytical procedures:

• Review the case to get an overview of the situation . A case can be only a few pages in length, however, it is most often very lengthy and contains a significant amount of detailed background information and statistics, with multilayered descriptions of the scenario, the roles and behaviors of various stakeholder groups, and situational events. Therefore, a quick reading of the case will help you gain an overall sense of the situation and illuminate the types of issues and problems that you will need to address in your paper. If your professor has provided questions intended to help frame your analysis, use them to guide your initial reading of the case.
• Read the case thoroughly . After gaining a general overview of the case, carefully read the content again with the purpose of understanding key circumstances, events, and behaviors among stakeholder groups. Look for information or data that appears contradictory, extraneous, or misleading. At this point, you should be taking notes as you read because this will help you develop a general outline of your paper. The aim is to obtain a complete understanding of the situation so that you can begin contemplating tentative answers to any questions your professor has provided or, if they have not provided, developing answers to your own questions about the case scenario and its connection to the course readings,lectures, and class discussions.
• Determine key stakeholder groups, issues, and events and the relationships they all have to each other . As you analyze the content, pay particular attention to identifying individuals, groups, or organizations described in the case and identify evidence of any problems or issues of concern that impact the situation in a negative way. Other things to look for include identifying any assumptions being made by or about each stakeholder, potential biased explanations or actions, explicit demands or ultimatums , and the underlying concerns that motivate these behaviors among stakeholders. The goal at this stage is to develop a comprehensive understanding of the situational and behavioral dynamics of the case and the explicit and implicit consequences of each of these actions.
• Identify the core problems . The next step in most case analysis assignments is to discern what the core [i.e., most damaging, detrimental, injurious] problems are within the organizational setting and to determine their implications. The purpose at this stage of preparing to write your analysis paper is to distinguish between the symptoms of core problems and the core problems themselves and to decide which of these must be addressed immediately and which problems do not appear critical but may escalate over time. Identify evidence from the case to support your decisions by determining what information or data is essential to addressing the core problems and what information is not relevant or is misleading.
• Explore alternative solutions . As noted, case analysis scenarios rarely have only one correct answer. Therefore, it is important to keep in mind that the process of analyzing the case and diagnosing core problems, while based on evidence, is a subjective process open to various avenues of interpretation. This means that you must consider alternative solutions or courses of action by critically examining strengths and weaknesses, risk factors, and the differences between short and long-term solutions. For each possible solution or course of action, consider the consequences they may have related to their implementation and how these recommendations might lead to new problems. Also, consider thinking about your recommended solutions or courses of action in relation to issues of fairness, equity, and inclusion.
• Decide on a final set of recommendations . The last stage in preparing to write a case analysis paper is to assert an opinion or viewpoint about the recommendations needed to help resolve the core problems as you see them and to make a persuasive argument for supporting this point of view. Prepare a clear rationale for your recommendations based on examining each element of your analysis. Anticipate possible obstacles that could derail their implementation. Consider any counter-arguments that could be made concerning the validity of your recommended actions. Finally, describe a set of criteria and measurable indicators that could be applied to evaluating the effectiveness of your implementation plan.

Use these steps as the framework for writing your paper. Remember that the more detailed you are in taking notes as you critically examine each element of the case, the more information you will have to draw from when you begin to write. This will save you time.

NOTE : If the process of preparing to write a case analysis paper is assigned as a student group project, consider having each member of the group analyze a specific element of the case, including drafting answers to the corresponding questions used by your professor to frame the analysis. This will help make the analytical process more efficient and ensure that the distribution of work is equitable. This can also facilitate who is responsible for drafting each part of the final case analysis paper and, if applicable, the in-class presentation.

Framework for Case Analysis . College of Management. University of Massachusetts; Hawes, Jon M. "Teaching is Not Telling: The Case Method as a Form of Interactive Learning." Journal for Advancement of Marketing Education 5 (Winter 2004): 47-54; Rasche, Christoph and Achim Seisreiner. Guidelines for Business Case Analysis . University of Potsdam; Writing a Case Study Analysis . University of Arizona Global Campus Writing Center; Van Ness, Raymond K. A Guide to Case Analysis . School of Business. State University of New York, Albany; Writing a Case Analysis . Business School, University of New South Wales.

Structure and Writing Style

A case analysis paper should be detailed, concise, persuasive, clearly written, and professional in tone and in the use of language . As with other forms of college-level academic writing, declarative statements that convey information, provide a fact, or offer an explanation or any recommended courses of action should be based on evidence. If allowed by your professor, any external sources used to support your analysis, such as course readings, should be properly cited under a list of references. The organization and structure of case analysis papers can vary depending on your professor’s preferred format, but its structure generally follows the steps used for analyzing the case.

Introduction

The introduction should provide a succinct but thorough descriptive overview of the main facts, issues, and core problems of the case . The introduction should also include a brief summary of the most relevant details about the situation and organizational setting. This includes defining the theoretical framework or conceptual model on which any questions were used to frame your analysis.

Following the rules of most college-level research papers, the introduction should then inform the reader how the paper will be organized. This includes describing the major sections of the paper and the order in which they will be presented. Unless you are told to do so by your professor, you do not need to preview your final recommendations in the introduction. U nlike most college-level research papers , the introduction does not include a statement about the significance of your findings because a case analysis assignment does not involve contributing new knowledge about a research problem.

Background Analysis

Background analysis can vary depending on any guiding questions provided by your professor and the underlying concept or theory that the case is based upon. In general, however, this section of your paper should focus on:

• Providing an overarching analysis of problems identified from the case scenario, including identifying events that stakeholders find challenging or troublesome,
• Identifying assumptions made by each stakeholder and any apparent biases they may exhibit,
• Describing any demands or claims made by or forced upon key stakeholders, and
• Highlighting any issues of concern or complaints expressed by stakeholders in response to those demands or claims.

These aspects of the case are often in the form of behavioral responses expressed by individuals or groups within the organizational setting. However, note that problems in a case situation can also be reflected in data [or the lack thereof] and in the decision-making, operational, cultural, or institutional structure of the organization. Additionally, demands or claims can be either internal and external to the organization [e.g., a case analysis involving a president considering arms sales to Saudi Arabia could include managing internal demands from White House advisors as well as demands from members of Congress].

Throughout this section, present all relevant evidence from the case that supports your analysis. Do not simply claim there is a problem, an assumption, a demand, or a concern; tell the reader what part of the case informed how you identified these background elements.

Identification of Problems

In most case analysis assignments, there are problems, and then there are problems . Each problem can reflect a multitude of underlying symptoms that are detrimental to the interests of the organization. The purpose of identifying problems is to teach students how to differentiate between problems that vary in severity, impact, and relative importance. Given this, problems can be described in three general forms: those that must be addressed immediately, those that should be addressed but the impact is not severe, and those that do not require immediate attention and can be set aside for the time being.

All of the problems you identify from the case should be identified in this section of your paper, with a description based on evidence explaining the problem variances. If the assignment asks you to conduct research to further support your assessment of the problems, include this in your explanation. Remember to cite those sources in a list of references. Use specific evidence from the case and apply appropriate concepts, theories, and models discussed in class or in relevant course readings to highlight and explain the key problems [or problem] that you believe must be solved immediately and describe the underlying symptoms and why they are so critical.

Alternative Solutions

This section is where you provide specific, realistic, and evidence-based solutions to the problems you have identified and make recommendations about how to alleviate the underlying symptomatic conditions impacting the organizational setting. For each solution, you must explain why it was chosen and provide clear evidence to support your reasoning. This can include, for example, course readings and class discussions as well as research resources, such as, books, journal articles, research reports, or government documents. In some cases, your professor may encourage you to include personal, anecdotal experiences as evidence to support why you chose a particular solution or set of solutions. Using anecdotal evidence helps promote reflective thinking about the process of determining what qualifies as a core problem and relevant solution .

Throughout this part of the paper, keep in mind the entire array of problems that must be addressed and describe in detail the solutions that might be implemented to resolve these problems.

Recommended Courses of Action

In some case analysis assignments, your professor may ask you to combine the alternative solutions section with your recommended courses of action. However, it is important to know the difference between the two. A solution refers to the answer to a problem. A course of action refers to a procedure or deliberate sequence of activities adopted to proactively confront a situation, often in the context of accomplishing a goal. In this context, proposed courses of action are based on your analysis of alternative solutions. Your description and justification for pursuing each course of action should represent the overall plan for implementing your recommendations.

For each course of action, you need to explain the rationale for your recommendation in a way that confronts challenges, explains risks, and anticipates any counter-arguments from stakeholders. Do this by considering the strengths and weaknesses of each course of action framed in relation to how the action is expected to resolve the core problems presented, the possible ways the action may affect remaining problems, and how the recommended action will be perceived by each stakeholder.

In addition, you should describe the criteria needed to measure how well the implementation of these actions is working and explain which individuals or groups are responsible for ensuring your recommendations are successful. In addition, always consider the law of unintended consequences. Outline difficulties that may arise in implementing each course of action and describe how implementing the proposed courses of action [either individually or collectively] may lead to new problems [both large and small].

Throughout this section, you must consider the costs and benefits of recommending your courses of action in relation to uncertainties or missing information and the negative consequences of success.

The conclusion should be brief and introspective. Unlike a research paper, the conclusion in a case analysis paper does not include a summary of key findings and their significance, a statement about how the study contributed to existing knowledge, or indicate opportunities for future research.

Begin by synthesizing the core problems presented in the case and the relevance of your recommended solutions. This can include an explanation of what you have learned about the case in the context of your answers to the questions provided by your professor. The conclusion is also where you link what you learned from analyzing the case with the course readings or class discussions. This can further demonstrate your understanding of the relationships between the practical case situation and the theoretical and abstract content of assigned readings and other course content.

Problems to Avoid

The literature on case analysis assignments often includes examples of difficulties students have with applying methods of critical analysis and effectively reporting the results of their assessment of the situation. A common reason cited by scholars is that the application of this type of teaching and learning method is limited to applied fields of social and behavioral sciences and, as a result, writing a case analysis paper can be unfamiliar to most students entering college.

After you have drafted your paper, proofread the narrative flow and revise any of these common errors:

• Unnecessary detail in the background section . The background section should highlight the essential elements of the case based on your analysis. Focus on summarizing the facts and highlighting the key factors that become relevant in the other sections of the paper by eliminating any unnecessary information.
• Analysis relies too much on opinion . Your analysis is interpretive, but the narrative must be connected clearly to evidence from the case and any models and theories discussed in class or in course readings. Any positions or arguments you make should be supported by evidence.
• Analysis does not focus on the most important elements of the case . Your paper should provide a thorough overview of the case. However, the analysis should focus on providing evidence about what you identify are the key events, stakeholders, issues, and problems. Emphasize what you identify as the most critical aspects of the case to be developed throughout your analysis. Be thorough but succinct.
• Writing is too descriptive . A paper with too much descriptive information detracts from your analysis of the complexities of the case situation. Questions about what happened, where, when, and by whom should only be included as essential information leading to your examination of questions related to why, how, and for what purpose.
• Inadequate definition of a core problem and associated symptoms . A common error found in case analysis papers is recommending a solution or course of action without adequately defining or demonstrating that you understand the problem. Make sure you have clearly described the problem and its impact and scope within the organizational setting. Ensure that you have adequately described the root causes w hen describing the symptoms of the problem.
• Recommendations lack specificity . Identify any use of vague statements and indeterminate terminology, such as, “A particular experience” or “a large increase to the budget.” These statements cannot be measured and, as a result, there is no way to evaluate their successful implementation. Provide specific data and use direct language in describing recommended actions.
• Unrealistic, exaggerated, or unattainable recommendations . Review your recommendations to ensure that they are based on the situational facts of the case. Your recommended solutions and courses of action must be based on realistic assumptions and fit within the constraints of the situation. Also note that the case scenario has already happened, therefore, any speculation or arguments about what could have occurred if the circumstances were different should be revised or eliminated.

Bee, Lian Song et al. "Business Students' Perspectives on Case Method Coaching for Problem-Based Learning: Impacts on Student Engagement and Learning Performance in Higher Education." Education & Training 64 (2022): 416-432; The Case Analysis . Fred Meijer Center for Writing and Michigan Authors. Grand Valley State University; Georgallis, Panikos and Kayleigh Bruijn. "Sustainability Teaching using Case-Based Debates." Journal of International Education in Business 15 (2022): 147-163; Hawes, Jon M. "Teaching is Not Telling: The Case Method as a Form of Interactive Learning." Journal for Advancement of Marketing Education 5 (Winter 2004): 47-54; Georgallis, Panikos, and Kayleigh Bruijn. "Sustainability Teaching Using Case-based Debates." Journal of International Education in Business 15 (2022): 147-163; .Dean,  Kathy Lund and Charles J. Fornaciari. "How to Create and Use Experiential Case-Based Exercises in a Management Classroom." Journal of Management Education 26 (October 2002): 586-603; Klebba, Joanne M. and Janet G. Hamilton. "Structured Case Analysis: Developing Critical Thinking Skills in a Marketing Case Course." Journal of Marketing Education 29 (August 2007): 132-137, 139; Klein, Norman. "The Case Discussion Method Revisited: Some Questions about Student Skills." Exchange: The Organizational Behavior Teaching Journal 6 (November 1981): 30-32; Mukherjee, Arup. "Effective Use of In-Class Mini Case Analysis for Discovery Learning in an Undergraduate MIS Course." The Journal of Computer Information Systems 40 (Spring 2000): 15-23; Pessoa, Silviaet al. "Scaffolding the Case Analysis in an Organizational Behavior Course: Making Analytical Language Explicit." Journal of Management Education 46 (2022): 226-251: Ramsey, V. J. and L. D. Dodge. "Case Analysis: A Structured Approach." Exchange: The Organizational Behavior Teaching Journal 6 (November 1981): 27-29; Schweitzer, Karen. "How to Write and Format a Business Case Study." ThoughtCo. https://www.thoughtco.com/how-to-write-and-format-a-business-case-study-466324 (accessed December 5, 2022); Reddy, C. D. "Teaching Research Methodology: Everything's a Case." Electronic Journal of Business Research Methods 18 (December 2020): 178-188; Volpe, Guglielmo. "Case Teaching in Economics: History, Practice and Evidence." Cogent Economics and Finance 3 (December 2015). doi:https://doi.org/10.1080/23322039.2015.1120977.

Writing Tip

Ca se Study and Case Analysis Are Not the Same!

Confusion often exists between what it means to write a paper that uses a case study research design and writing a paper that analyzes a case; they are two different types of approaches to learning in the social and behavioral sciences. Professors as well as educational researchers contribute to this confusion because they often use the term "case study" when describing the subject of analysis for a case analysis paper. But you are not studying a case for the purpose of generating a comprehensive, multi-faceted understanding of a research problem. R ather, you are critically analyzing a specific scenario to argue logically for recommended solutions and courses of action that lead to optimal outcomes applicable to professional practice.

To avoid any confusion, here are twelve characteristics that delineate the differences between writing a paper using the case study research method and writing a case analysis paper:

• Case study is a method of in-depth research and rigorous inquiry ; case analysis is a reliable method of teaching and learning . A case study is a modality of research that investigates a phenomenon for the purpose of creating new knowledge, solving a problem, or testing a hypothesis using empirical evidence derived from the case being studied. Often, the results are used to generalize about a larger population or within a wider context. The writing adheres to the traditional standards of a scholarly research study. A case analysis is a pedagogical tool used to teach students how to reflect and think critically about a practical, real-life problem in an organizational setting.
• The researcher is responsible for identifying the case to study; a case analysis is assigned by your professor . As the researcher, you choose the case study to investigate in support of obtaining new knowledge and understanding about the research problem. The case in a case analysis assignment is almost always provided, and sometimes written, by your professor and either given to every student in class to analyze individually or to a small group of students, or students select a case to analyze from a predetermined list.
• A case study is indeterminate and boundless; a case analysis is predetermined and confined . A case study can be almost anything [see item 9 below] as long as it relates directly to examining the research problem. This relationship is the only limit to what a researcher can choose as the subject of their case study. The content of a case analysis is determined by your professor and its parameters are well-defined and limited to elucidating insights of practical value applied to practice.
• Case study is fact-based and describes actual events or situations; case analysis can be entirely fictional or adapted from an actual situation . The entire content of a case study must be grounded in reality to be a valid subject of investigation in an empirical research study. A case analysis only needs to set the stage for critically examining a situation in practice and, therefore, can be entirely fictional or adapted, all or in-part, from an actual situation.
• Research using a case study method must adhere to principles of intellectual honesty and academic integrity; a case analysis scenario can include misleading or false information . A case study paper must report research objectively and factually to ensure that any findings are understood to be logically correct and trustworthy. A case analysis scenario may include misleading or false information intended to deliberately distract from the central issues of the case. The purpose is to teach students how to sort through conflicting or useless information in order to come up with the preferred solution. Any use of misleading or false information in academic research is considered unethical.
• Case study is linked to a research problem; case analysis is linked to a practical situation or scenario . In the social sciences, the subject of an investigation is most often framed as a problem that must be researched in order to generate new knowledge leading to a solution. Case analysis narratives are grounded in real life scenarios for the purpose of examining the realities of decision-making behavior and processes within organizational settings. A case analysis assignments include a problem or set of problems to be analyzed. However, the goal is centered around the act of identifying and evaluating courses of action leading to best possible outcomes.
• The purpose of a case study is to create new knowledge through research; the purpose of a case analysis is to teach new understanding . Case studies are a choice of methodological design intended to create new knowledge about resolving a research problem. A case analysis is a mode of teaching and learning intended to create new understanding and an awareness of uncertainty applied to practice through acts of critical thinking and reflection.
• A case study seeks to identify the best possible solution to a research problem; case analysis can have an indeterminate set of solutions or outcomes . Your role in studying a case is to discover the most logical, evidence-based ways to address a research problem. A case analysis assignment rarely has a single correct answer because one of the goals is to force students to confront the real life dynamics of uncertainly, ambiguity, and missing or conflicting information within professional practice. Under these conditions, a perfect outcome or solution almost never exists.
• Case study is unbounded and relies on gathering external information; case analysis is a self-contained subject of analysis . The scope of a case study chosen as a method of research is bounded. However, the researcher is free to gather whatever information and data is necessary to investigate its relevance to understanding the research problem. For a case analysis assignment, your professor will often ask you to examine solutions or recommended courses of action based solely on facts and information from the case.
• Case study can be a person, place, object, issue, event, condition, or phenomenon; a case analysis is a carefully constructed synopsis of events, situations, and behaviors . The research problem dictates the type of case being studied and, therefore, the design can encompass almost anything tangible as long as it fulfills the objective of generating new knowledge and understanding. A case analysis is in the form of a narrative containing descriptions of facts, situations, processes, rules, and behaviors within a particular setting and under a specific set of circumstances.
• Case study can represent an open-ended subject of inquiry; a case analysis is a narrative about something that has happened in the past . A case study is not restricted by time and can encompass an event or issue with no temporal limit or end. For example, the current war in Ukraine can be used as a case study of how medical personnel help civilians during a large military conflict, even though circumstances around this event are still evolving. A case analysis can be used to elicit critical thinking about current or future situations in practice, but the case itself is a narrative about something finite and that has taken place in the past.
• Multiple case studies can be used in a research study; case analysis involves examining a single scenario . Case study research can use two or more cases to examine a problem, often for the purpose of conducting a comparative investigation intended to discover hidden relationships, document emerging trends, or determine variations among different examples. A case analysis assignment typically describes a stand-alone, self-contained situation and any comparisons among cases are conducted during in-class discussions and/or student presentations.

The Case Analysis . Fred Meijer Center for Writing and Michigan Authors. Grand Valley State University; Mills, Albert J. , Gabrielle Durepos, and Eiden Wiebe, editors. Encyclopedia of Case Study Research . Thousand Oaks, CA: SAGE Publications, 2010; Ramsey, V. J. and L. D. Dodge. "Case Analysis: A Structured Approach." Exchange: The Organizational Behavior Teaching Journal 6 (November 1981): 27-29; Yin, Robert K. Case Study Research and Applications: Design and Methods . 6th edition. Thousand Oaks, CA: Sage, 2017; Crowe, Sarah et al. “The Case Study Approach.” BMC Medical Research Methodology 11 (2011):  doi: 10.1186/1471-2288-11-100; Yin, Robert K. Case Study Research: Design and Methods . 4th edition. Thousand Oaks, CA: Sage Publishing; 1994.

• << Previous: Reviewing Collected Works
• Next: Writing a Case Study >>
• Last Updated: Mar 6, 2024 1:00 PM
• URL: https://libguides.usc.edu/writingguide/assignments

Browser does not support script.

• Departments and Institutes
• Research centres and groups
• Chair's Blog: Summer Term 2022
• Staff wellbeing

Case studies

Case studies usually involve real-life situations and often take the form of a problem-based inquiry approach; in other words students are presented with a complex real life situation that they are asked to find a solution to. “The benefits of utilizing case studies in instruction include the way that cases model how to think professionally about real problems and situations, helping candidates to think productively about concrete experiences” (Kleinfeld, 1990 in Ulanoff, Fingon and Beltran, 2009). The case study method involves placing students in the role of decision-makers and asking them to address a challenge that may confront a company, non-profit organisation or government department. In the absence of a single straightforward answer students are expected to exchange ideas, consider possible theoretical explanations and data, and weigh up possible solutions. Based on this exchange and evaluation of mixed data they are expected to come up with a decision, and choose a solution to the particular challenge. Though case study learning and assessment may take many forms the common thread is that the case study involves a real-life situation and finding solutions is the focus of the assessment.

Advantages of case studies

• Enables students to apply their knowledge and skills to real life situations.
• Can be undertaken individually or as a group assessment.
• Generally designed to assess the higher levels of Bloom’s taxonomy of educational objectives (application, analysis and evaluation).
• Well adapted to multi- or inter-disciplinary learning.
• Calls on students to demonstrate a range of different skills such as the selection on information, analysis, decision-making problem-solving and presentation.
• In the case of a group-based approach students are given the opportunity to demonstrate their ability to collaborate and communicate effectively.
• Supports the development of a range of valuable employability skills which are likely to be attractive to employers and students alike.

Challenges of case studies

• Case studies can be used in time-constrained examinations but this method of assessment really lends itself better to a coursework approach.
• Can be a complex activity that involves negotiating a range of media that may be hard to contain in a controlled environment.
• It is important to have realistic expectations of what actually can be achieved.
• Planning and preparing for case study work can be time-consuming for teachers.

How students might experience case studies

There is some evidence to suggest that case studies increase students’ motivation. Students are often very interested in working on real life situations. It brings their learning alive and enables them not only to develop solutions to actual situations/problems but also to understand in new ways the valuable role that theory and relevant concepts can play as part of this process. In addition as part of their work on the case study they are clearly developing valuable transferable skills that they can take forward into the workplace and society at large. Students may not be used to this form of assessment so they will need clear guidance as to what is expected (length, format, main elements), a clear explanation of marking criteria as well as development in the different skills they will need to acquire in order to successfully complete the case study. These will in part depend on the nature of the case study - is data analysis involved?; where and how will students find relevant qualitative and quantitative data?; what is the appropriate way of citing and referencing?

Reliability, validity, fairness and inclusivity of case studies

Teaching and learning activities should be carefully designed to support the work on the case study or the development of the relevant skills and knowledge bases. From an inclusive design perspective case studies are an attractive form of learning and assessment.  Depending on the nature of the inquiry students may be given a degree of choice over their case study and thus be in a position to bring their different backgrounds and experience to bear. In any case, it is important to ensure that the chosen case studies are accessible to all students taking the course. In the case of first year students the teacher may want to provide all the relevant materials to the students. For more advanced students, they may be expected to do some research and to identify relevant supporting materials for the case study inquiry. Where group work is involved a number of options may be considered to ensure fairness. The students may complete some elements of both formative and summative work as a group as well as others individually. For example, students may complete various tasks or give a presentation on the case study as a group but write up part of the final case study individually. In addition, it is relatively common practice to ask students engaged in groupwork to write a short reflective piece discussing their experience of group work. Students can also be asked to rate their contribution and the contribution of other members of the group using one of a number of online group assessment tools such as WebPA and Teammates.

How to maintain and ensure rigour in case studies

Critical to ensuring rigour is having clarity about the different parts of the case study or, in the case of a single assessment task, the criteria against which the assessment will be marked; the weight that will be attached to different parts of the assignment, and the marking scheme.  Marking and moderation should follow departmental practice.

How to limit possible misconduct in case studies

Whether the students are working in groups or individually teachers can check that the work is the work of particular students by designing in opportunities to assess (formatively or summatively) work at several points in the assessment process. This can be done by asking students to present work in written or oral form – either by submitting assignment tasks via Moodle or making short presentations in class. In addition to serving as a check for misconduct this also provides an opportunity for teachers and peers to give constructive feedback on the development of the case study and as such constitutes good practice.

LSE examples

Daniel Ferreira discussed his use of case studies in teaching Master’s level Finance students for many years, and, starting in 2016/17 undergraduates with the introduction of the Finance department’s new BSc programme

http://lti.lse.ac.uk/lse-innovators/irene-papanicolas-healthy-collaboration/

Further resources

University of New South Wales, Sydney: Assessment by Case Studies and Scenarios https://teaching.unsw.edu.au/assessment-case-studies-and-scenarios

Assessment Resources at Hong Kong University: Types of Assessment Methods: Case Study http://ar.cetl.hku.hk/am_case_study.htm

Bonney, K.M. (2015) Case Study Teaching Method Improves Student Performance and Perceptions of Learning Gains.  Journal of Microbiological Education , 16(1): 21–28

Ulanoff, S.H., Fingon, J.C. and Beltrán, D. (2009) Using Case Studies To Assess Candidates’ Knowledge and Skills in a Graduate Reading Program,  Teacher Education Quarterly,  6(2): 125-142

Fry, H., Ketteridge, S. and Marshall, S. (1999)  A Handbook for Teaching and Learning in Higher Education,  Routledge, UK

Back to Assessment methods

Back to Toolkit Main page

Contact your Eden Centre departmental adviser

If you have any suggestions for future Toolkit development, get in touch using our email below!

Email: [email protected].

Why Do Employers Use Case Studies at Assessment Centers?

What to expect from a case study exercise, how to prepare for the case study exercise in 2024, how to approach a group exercise, how to approach a presentation, case study exercises at assessment centers (2024 guide).

Updated November 21, 2023

Should you be invited to be tested at an assessment center as part of an employer's recruitment process, one of the exercises you may face is a case study .

A case study exercise presents you with a scenario similar to what you would experience in the job you have applied for.

It will generally be accompanied by documents, emails or other forms of information.

You are asked to make business decisions based on the data you have been provided with, either alone or as part of a group of candidates.

A case study enables employers to assess your skill-base and likely performance in the job, providing them with a more rounded view of the type of employee you would be and the value you would bring to the company.

Commonly used in the finance, banking, legal and business management industries, the main advantage to employers of using case study exercises is to see candidates in action, demonstrating the skills they would be expected to use at work.

The skills assessed when participating in a case study exercise will vary depending on the employer, the industry and the job applied for, but may include:

• Analytical skills
• Strategic thinking
• Decision making
• Problem-solving
• Communication
• Stress tolerance
• The ability to assimilate information quickly and effectively
• Organisational skills
• Situational judgment
• Commercial awareness
• Time management
• Team working
• Knowledge pertinent to the industry or job, for example, marketing skills

Despite the skills that the employer is actively assessing, such as those mentioned above, success in a case study exercise relies on your ability to:

• Interpret and analyze the information provided
• Reach a decision
• Use commercial awareness
• Manage your time
• Communicate well

Practice Case Study Exercises with JobTestPrep

There are generally two types of case study exercise that you may face as part of a selection process:

• Subject-related case studies pertinent to the job you are applying for and the related industry
• General case studies that assess your overall aptitude and skills

The actual scenario of the case study exercise you face will vary, but examples of typical case studies include:

• Expanding a team or department
• Deciding whether an acquisition or merger is advisable
• Investigating whether to begin a new product line
• Re-organisation of management structure
• The creation of an advertising campaign
• Responding to negative publicity
• Choosing from three business proposals
• Developing a social media presence

Prepare for Case Study Exercises with JobTestPrep

For example: You are presented with the scenario of an IT company that went through an expensive re-brand one year ago. At that time, the company moved to bigger premises in a better area, and two new teams of developers were recruited to work with two new clients. The IT company has recently lost one of those clients and is facing increasing costs as the rent is raised for their premises. The company's directors have concluded that they must make one of the following changes: Make staff redundancies and offer the chance to several employees to change to part-time hours Move to less expensive premises in a less desirable area Combine a move to a flexible working business model where employees work part of the week from home and desk-share in the office along with a physical move to smaller premises in the same area where the IT company is currently based

You are asked to advise the directors on which change would provide the greatest benefit.

Here is another example:

A multi-national environmental testing organization buys out an oil-testing laboratory. A gap test is carried out on whether: The oil-testing lab should be brought in line with the rest of the organization concerning its processes, customer interface, and testing procedures The oil-testing lab should be closed down and its clients absorbed into the rest of the organization The oil-testing lab should be allowed to continue as it is, but new processes put in place between it and the larger organization

You are asked to consider the findings of the gap test and suggest the best course of action.

Just as you would prepare before a job interview, it is always in your best interests to prepare before facing a case study exercise at an assessment center.

Step 1 . Do the Research

There is a whole range of research you can look into to prepare yourself for the case study exercise:

• The job description and any other literature or documents forwarded to you
• The employer's website and social media
• Industry related news stories and developments

Any of the above should provide you with a better understanding of the job you have applied for, the industry you will work within, and the culture and values of the employer.

Step 2 . Use Practice Case Studies

Practicing case study exercises in the run-up to the assessment day is one of the best ways you can prepare for the real thing.

Unless the employer provides sample case studies on their website or as part of their recruitment pack, you will not know the exact format that the exercise will take; however, you can build familiarity with the overall process of a case study through practice.

You can find plenty of practice case study exercises online. Most of these come at a cost, but you may also be able to find free sample case studies too.

For case study resources at a cost, have a look at JobTestPrep .

For two free sample case study exercises, you might like to visit Bain & Company's website .

Scroll down to the Associate Consultant Case Library. Europa also offers an extensive and detailed sample case study .

Step 3 . Timed Practice

Once you have sourced one or more practice case studies, take the opportunity to practice to a time limit.

The case study may come with a time limit, or the employer may have already told you how long you will have to complete the real case study exercise on the day.

Alternatively, set your reasonable time limit.

Timed practice will improve your response time and explain exactly how much time you should allocate to each stage of the case study process.

Step 4 . Improve Your Reading Comprehension

One skill that is key to handle a case study exercise successfully is your reading comprehension, that is, your ability to understand written information, interpret it and describe it in your own words.

In the context of a case study, this skill will help you to assimilate the information provided to you quickly, analyze it and ultimately reach a decision.

In the run-up to your assessment day, put aside time to improve your reading comprehension by reading a wide variety of material and picking out the key points of each passage.

You might find it especially helpful to read professional journals and news articles related to the job you have applied for and the related industry.

Try to improve the speed at which you can read but still retain information too. This will prove helpful during the real case study exercise.

Step 5 . Practice Mental Math

The case study exercise may include prices, area measurements, staff numbers, salaries and other numeric values.

It is important that you can complete basic mental math calculations, such as multiplication and percentages.

Practice your mental math using puzzle books, online math resources and math problems that you create yourself.

You can find plenty of online business math resources, for example:

• The University of Alabama at Birmingham Math and Business Guide
• Money Instructor
• Open Textbook Library

If you need to prepare for a number of different employment tests and want to outsmart the competition, choose a Premium Membership from JobTestPrep . You will get access to three PrepPacks of your choice, from a database that covers all the major test providers and employers and tailored profession packs.

Get a Premium Package Now

Top Tips for Approaching Case Study Exercises

Now that you have prepared yourself, you can further improve your chances of a successful outcome by following our top tips on approaching case study exercises on the day.

Read the Information Carefully

Read all of the information provided as part of your case study exercise to understand what is being asked of you fully.

Quickly identify the key points in the task and the overall decision you have been asked to make, for example:

• Has the exercise provided you with a choice of outcomes you must decide between, or must you create the outcome yourself?
• What information do you need to make your decision?
• Are there calculations involved in the task?
• What character are you playing in the task (for example, HR manager or business consultant) and what are that character's motivations?
• Who is your character presenting their response to? Company directors, client or HR department?

Prioritize the Information

Prioritize the information by importance.

Which pieces of information are most pertinent to the task, and what key data do they provide?

Can any of the information be dismissed? Does any of the information contradict or sit in conflict with others?

Divide Up the Tasks and Allocate Time

You will generally be asked to come to a conclusion or advise a course of action regarding your case study exercise; however, you may have to carry out several tasks to arrive at this result.

Once you have read through the information, plan out what tasks the exercise will entail and allocate time for each one.

Do Not Be Distracted by Finding the Only 'Right' Answer

Where you are provided with several outcomes, and you must decide on one, do not assume that anyone's outcome is the only right answer to give.

It may be that any of the outcomes could be correct if you can sufficiently support your decision from the information provided.

Keep the Objective in Focus

• What does the task ask you to do?
• Must you choose between three business acquisitions?
• Are you providing advice on whether or not to invest?
• Are you putting together a plan for a staff redundancy situation?

Keep the objective of the case study exercise in mind at all times.

Support Your Decision With Evidence

The conclusion you come to may seem obvious to you, but you must be able to support your decision with evidence.

Why would it be better for the company to invest in property overstock? What is the benefit to the company of entering a new market?

It is not sufficient to know which outcome would be the best. As in the real-life business world, you must be able to support your claims.

If you are assessed as part of a group, you must arrive at a conclusion as a team and bear in mind your strengths.

For example, do you have a good eye for detail and would therefore be suited to the analytical part of the task?

Arrive at a list of tasks together and then assign the tasks to different members of the group.

Please make sure you contribute to the group discussions but do not dominate them.

Group assessments are generally used by employers who place value on leadership, teamwork and communication skills.

If you are asked to present your findings or conclusion as part of a case study exercise, bear in mind to whom the task has asked you to make that presentation.

For example, a business client or a marketing manager.

Make sure that you can fully support the reasons that you came to your conclusion.

If you are presenting as a group, make sure that each group member has their role to play in the presentation and that everyone knows why the group came to that conclusion.

Act professionally to suit the job you have applied for. Be polite, confident and well-spoken.

Case study exercises are just one of the many methods that employers use to assess job applicants, and as with any other aspect of the selection process, they require a degree of consideration and preparation.

The best way to improve your chances of a successful outcome and reduce exam tension is to research the job and the industry, practice case study exercises and improve your skills.

You might also be interested in these other Psychometric Success articles:

Or explore the Aptitude Tests / Test Types sections.

Do Your Students Know How to Analyze a Case—Really?

Explore more.

• Case Teaching
• Student Engagement

J ust as actors, athletes, and musicians spend thousands of hours practicing their craft, business students benefit from practicing their critical-thinking and decision-making skills. Students, however, often have limited exposure to real-world problem-solving scenarios; they need more opportunities to practice tackling tough business problems and deciding on—and executing—the best solutions.

To ensure students have ample opportunity to develop these critical-thinking and decision-making skills, we believe business faculty should shift from teaching mostly principles and ideas to mostly applications and practices. And in doing so, they should emphasize the case method, which simulates real-world management challenges and opportunities for students.

To help educators facilitate this shift and help students get the most out of case-based learning, we have developed a framework for analyzing cases. We call it PACADI (Problem, Alternatives, Criteria, Analysis, Decision, Implementation); it can improve learning outcomes by helping students better solve and analyze business problems, make decisions, and develop and implement strategy. Here, we’ll explain why we developed this framework, how it works, and what makes it an effective learning tool.

The Case for Cases: Helping Students Think Critically

Business students must develop critical-thinking and analytical skills, which are essential to their ability to make good decisions in functional areas such as marketing, finance, operations, and information technology, as well as to understand the relationships among these functions. For example, the decisions a marketing manager must make include strategic planning (segments, products, and channels); execution (digital messaging, media, branding, budgets, and pricing); and operations (integrated communications and technologies), as well as how to implement decisions across functional areas.

Faculty can use many types of cases to help students develop these skills. These include the prototypical “paper cases”; live cases , which feature guest lecturers such as entrepreneurs or corporate leaders and on-site visits; and multimedia cases , which immerse students into real situations. Most cases feature an explicit or implicit decision that a protagonist—whether it is an individual, a group, or an organization—must make.

For students new to learning by the case method—and even for those with case experience—some common issues can emerge; these issues can sometimes be a barrier for educators looking to ensure the best possible outcomes in their case classrooms. Unsure of how to dig into case analysis on their own, students may turn to the internet or rely on former students for “answers” to assigned cases. Or, when assigned to provide answers to assignment questions in teams, students might take a divide-and-conquer approach but not take the time to regroup and provide answers that are consistent with one other.

To help address these issues, which we commonly experienced in our classes, we wanted to provide our students with a more structured approach for how they analyze cases—and to really think about making decisions from the protagonists’ point of view. We developed the PACADI framework to address this need.

PACADI: A Six-Step Decision-Making Approach

The PACADI framework is a six-step decision-making approach that can be used in lieu of traditional end-of-case questions. It offers a structured, integrated, and iterative process that requires students to analyze case information, apply business concepts to derive valuable insights, and develop recommendations based on these insights.

Prior to beginning a PACADI assessment, which we’ll outline here, students should first prepare a two-paragraph summary—a situation analysis—that highlights the key case facts. Then, we task students with providing a five-page PACADI case analysis (excluding appendices) based on the following six steps.

Step 1: Problem definition. What is the major challenge, problem, opportunity, or decision that has to be made? If there is more than one problem, choose the most important one. Often when solving the key problem, other issues will surface and be addressed. The problem statement may be framed as a question; for example, How can brand X improve market share among millennials in Canada? Usually the problem statement has to be re-written several times during the analysis of a case as students peel back the layers of symptoms or causation.

Step 2: Alternatives. Identify in detail the strategic alternatives to address the problem; three to five options generally work best. Alternatives should be mutually exclusive, realistic, creative, and feasible given the constraints of the situation. Doing nothing or delaying the decision to a later date are not considered acceptable alternatives.

Step 3: Criteria. What are the key decision criteria that will guide decision-making? In a marketing course, for example, these may include relevant marketing criteria such as segmentation, positioning, advertising and sales, distribution, and pricing. Financial criteria useful in evaluating the alternatives should be included—for example, income statement variables, customer lifetime value, payback, etc. Students must discuss their rationale for selecting the decision criteria and the weights and importance for each factor.

Step 4: Analysis. Provide an in-depth analysis of each alternative based on the criteria chosen in step three. Decision tables using criteria as columns and alternatives as rows can be helpful. The pros and cons of the various choices as well as the short- and long-term implications of each may be evaluated. Best, worst, and most likely scenarios can also be insightful.

Step 5: Decision. Students propose their solution to the problem. This decision is justified based on an in-depth analysis. Explain why the recommendation made is the best fit for the criteria.

Step 6: Implementation plan. Sound business decisions may fail due to poor execution. To enhance the likeliness of a successful project outcome, students describe the key steps (activities) to implement the recommendation, timetable, projected costs, expected competitive reaction, success metrics, and risks in the plan.

“Students note that using the PACADI framework yields ‘aha moments’—they learned something surprising in the case that led them to think differently about the problem and their proposed solution.”

PACADI’s Benefits: Meaningfully and Thoughtfully Applying Business Concepts

The PACADI framework covers all of the major elements of business decision-making, including implementation, which is often overlooked. By stepping through the whole framework, students apply relevant business concepts and solve management problems via a systematic, comprehensive approach; they’re far less likely to surface piecemeal responses.

As students explore each part of the framework, they may realize that they need to make changes to a previous step. For instance, when working on implementation, students may realize that the alternative they selected cannot be executed or will not be profitable, and thus need to rethink their decision. Or, they may discover that the criteria need to be revised since the list of decision factors they identified is incomplete (for example, the factors may explain key marketing concerns but fail to address relevant financial considerations) or is unrealistic (for example, they suggest a 25 percent increase in revenues without proposing an increased promotional budget).

In addition, the PACADI framework can be used alongside quantitative assignments, in-class exercises, and business and management simulations. The structured, multi-step decision framework encourages careful and sequential analysis to solve business problems. Incorporating PACADI as an overarching decision-making method across different projects will ultimately help students achieve desired learning outcomes. As a practical “beyond-the-classroom” tool, the PACADI framework is not a contrived course assignment; it reflects the decision-making approach that managers, executives, and entrepreneurs exercise daily. Case analysis introduces students to the real-world process of making business decisions quickly and correctly, often with limited information. This framework supplies an organized and disciplined process that students can readily defend in writing and in class discussions.

PACADI in Action: An Example

Here’s an example of how students used the PACADI framework for a recent case analysis on CVS, a large North American drugstore chain.

The CVS Prescription for Customer Value*

PACADI Stage

Summary Response

How should CVS Health evolve from the “drugstore of your neighborhood” to the “drugstore of your future”?

Alternatives

A1. Kaizen (continuous improvement)

A2. Product development

A3. Market development

A4. Personalization (micro-targeting)

Criteria (include weights)

C1. Customer value: service, quality, image, and price (40%)

C2. Customer obsession (20%)

C3. Growth through related businesses (20%)

C4. Customer retention and customer lifetime value (20%)

Each alternative was analyzed by each criterion using a Customer Value Assessment Tool

Alternative 4 (A4): Personalization was selected. This is operationalized via: segmentation—move toward segment-of-1 marketing; geodemographics and lifestyle emphasis; predictive data analysis; relationship marketing; people, principles, and supply chain management; and exceptional customer service.

Implementation

Partner with leading medical school

Curbside pick-up

Pet pharmacy

E-newsletter for customers and employees

Employee incentive program

CVS beauty days

Expand to Latin America and Caribbean

Healthier/happier corner

Holiday toy drives/community outreach

*Source: A. Weinstein, Y. Rodriguez, K. Sims, R. Vergara, “The CVS Prescription for Superior Customer Value—A Case Study,” Back to the Future: Revisiting the Foundations of Marketing from Society for Marketing Advances, West Palm Beach, FL (November 2, 2018).

Results of Using the PACADI Framework

When faculty members at our respective institutions at Nova Southeastern University (NSU) and the University of North Carolina Wilmington have used the PACADI framework, our classes have been more structured and engaging. Students vigorously debate each element of their decision and note that this framework yields an “aha moment”—they learned something surprising in the case that led them to think differently about the problem and their proposed solution.

These lively discussions enhance individual and collective learning. As one external metric of this improvement, we have observed a 2.5 percent increase in student case grade performance at NSU since this framework was introduced.

Tips to Get Started

The PACADI approach works well in in-person, online, and hybrid courses. This is particularly important as more universities have moved to remote learning options. Because students have varied educational and cultural backgrounds, work experience, and familiarity with case analysis, we recommend that faculty members have students work on their first case using this new framework in small teams (two or three students). Additional analyses should then be solo efforts.

To use PACADI effectively in your classroom, we suggest the following:

Advise your students that your course will stress critical thinking and decision-making skills, not just course concepts and theory.

Use a varied mix of case studies. As marketing professors, we often address consumer and business markets; goods, services, and digital commerce; domestic and global business; and small and large companies in a single MBA course.

As a starting point, provide a short explanation (about 20 to 30 minutes) of the PACADI framework with a focus on the conceptual elements. You can deliver this face to face or through videoconferencing.

Give students an opportunity to practice the case analysis methodology via an ungraded sample case study. Designate groups of five to seven students to discuss the case and the six steps in breakout sessions (in class or via Zoom).

Ensure case analyses are weighted heavily as a grading component. We suggest 30–50 percent of the overall course grade.

Once cases are graded, debrief with the class on what they did right and areas needing improvement (30- to 40-minute in-person or Zoom session).

Encourage faculty teams that teach common courses to build appropriate instructional materials, grading rubrics, videos, sample cases, and teaching notes.

When selecting case studies, we have found that the best ones for PACADI analyses are about 15 pages long and revolve around a focal management decision. This length provides adequate depth yet is not protracted. Some of our tested and favorite marketing cases include Brand W , Hubspot , Kraft Foods Canada , TRSB(A) , and Whiskey & Cheddar .

Art Weinstein , Ph.D., is a professor of marketing at Nova Southeastern University, Fort Lauderdale, Florida. He has published more than 80 scholarly articles and papers and eight books on customer-focused marketing strategy. His latest book is Superior Customer Value—Finding and Keeping Customers in the Now Economy . Dr. Weinstein has consulted for many leading technology and service companies.

Herbert V. Brotspies , D.B.A., is an adjunct professor of marketing at Nova Southeastern University. He has over 30 years’ experience as a vice president in marketing, strategic planning, and acquisitions for Fortune 50 consumer products companies working in the United States and internationally. His research interests include return on marketing investment, consumer behavior, business-to-business strategy, and strategic planning.

John T. Gironda , Ph.D., is an assistant professor of marketing at the University of North Carolina Wilmington. His research has been published in Industrial Marketing Management, Psychology & Marketing , and Journal of Marketing Management . He has also presented at major marketing conferences including the American Marketing Association, Academy of Marketing Science, and Society for Marketing Advances.

Related Articles

We use cookies to understand how you use our site and to improve your experience, including personalizing content. Learn More . By continuing to use our site, you accept our use of cookies and revised Privacy Policy .

• Privacy Policy

Home » Case Study – Methods, Examples and Guide

Case Study – Methods, Examples and Guide

Table of Contents

A case study is a research method that involves an in-depth examination and analysis of a particular phenomenon or case, such as an individual, organization, community, event, or situation.

It is a qualitative research approach that aims to provide a detailed and comprehensive understanding of the case being studied. Case studies typically involve multiple sources of data, including interviews, observations, documents, and artifacts, which are analyzed using various techniques, such as content analysis, thematic analysis, and grounded theory. The findings of a case study are often used to develop theories, inform policy or practice, or generate new research questions.

Types of Case Study

Types and Methods of Case Study are as follows:

Single-Case Study

A single-case study is an in-depth analysis of a single case. This type of case study is useful when the researcher wants to understand a specific phenomenon in detail.

For Example , A researcher might conduct a single-case study on a particular individual to understand their experiences with a particular health condition or a specific organization to explore their management practices. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of a single-case study are often used to generate new research questions, develop theories, or inform policy or practice.

Multiple-Case Study

A multiple-case study involves the analysis of several cases that are similar in nature. This type of case study is useful when the researcher wants to identify similarities and differences between the cases.

For Example, a researcher might conduct a multiple-case study on several companies to explore the factors that contribute to their success or failure. The researcher collects data from each case, compares and contrasts the findings, and uses various techniques to analyze the data, such as comparative analysis or pattern-matching. The findings of a multiple-case study can be used to develop theories, inform policy or practice, or generate new research questions.

Exploratory Case Study

An exploratory case study is used to explore a new or understudied phenomenon. This type of case study is useful when the researcher wants to generate hypotheses or theories about the phenomenon.

For Example, a researcher might conduct an exploratory case study on a new technology to understand its potential impact on society. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as grounded theory or content analysis. The findings of an exploratory case study can be used to generate new research questions, develop theories, or inform policy or practice.

Descriptive Case Study

A descriptive case study is used to describe a particular phenomenon in detail. This type of case study is useful when the researcher wants to provide a comprehensive account of the phenomenon.

For Example, a researcher might conduct a descriptive case study on a particular community to understand its social and economic characteristics. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of a descriptive case study can be used to inform policy or practice or generate new research questions.

Instrumental Case Study

An instrumental case study is used to understand a particular phenomenon that is instrumental in achieving a particular goal. This type of case study is useful when the researcher wants to understand the role of the phenomenon in achieving the goal.

For Example, a researcher might conduct an instrumental case study on a particular policy to understand its impact on achieving a particular goal, such as reducing poverty. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of an instrumental case study can be used to inform policy or practice or generate new research questions.

Case Study Data Collection Methods

Here are some common data collection methods for case studies:

Interviews involve asking questions to individuals who have knowledge or experience relevant to the case study. Interviews can be structured (where the same questions are asked to all participants) or unstructured (where the interviewer follows up on the responses with further questions). Interviews can be conducted in person, over the phone, or through video conferencing.

Observations

Observations involve watching and recording the behavior and activities of individuals or groups relevant to the case study. Observations can be participant (where the researcher actively participates in the activities) or non-participant (where the researcher observes from a distance). Observations can be recorded using notes, audio or video recordings, or photographs.

Documents can be used as a source of information for case studies. Documents can include reports, memos, emails, letters, and other written materials related to the case study. Documents can be collected from the case study participants or from public sources.

Surveys involve asking a set of questions to a sample of individuals relevant to the case study. Surveys can be administered in person, over the phone, through mail or email, or online. Surveys can be used to gather information on attitudes, opinions, or behaviors related to the case study.

Artifacts are physical objects relevant to the case study. Artifacts can include tools, equipment, products, or other objects that provide insights into the case study phenomenon.

How to conduct Case Study Research

Conducting a case study research involves several steps that need to be followed to ensure the quality and rigor of the study. Here are the steps to conduct case study research:

• Define the research questions: The first step in conducting a case study research is to define the research questions. The research questions should be specific, measurable, and relevant to the case study phenomenon under investigation.
• Select the case: The next step is to select the case or cases to be studied. The case should be relevant to the research questions and should provide rich and diverse data that can be used to answer the research questions.
• Collect data: Data can be collected using various methods, such as interviews, observations, documents, surveys, and artifacts. The data collection method should be selected based on the research questions and the nature of the case study phenomenon.
• Analyze the data: The data collected from the case study should be analyzed using various techniques, such as content analysis, thematic analysis, or grounded theory. The analysis should be guided by the research questions and should aim to provide insights and conclusions relevant to the research questions.
• Draw conclusions: The conclusions drawn from the case study should be based on the data analysis and should be relevant to the research questions. The conclusions should be supported by evidence and should be clearly stated.
• Validate the findings: The findings of the case study should be validated by reviewing the data and the analysis with participants or other experts in the field. This helps to ensure the validity and reliability of the findings.
• Write the report: The final step is to write the report of the case study research. The report should provide a clear description of the case study phenomenon, the research questions, the data collection methods, the data analysis, the findings, and the conclusions. The report should be written in a clear and concise manner and should follow the guidelines for academic writing.

Examples of Case Study

Here are some examples of case study research:

• The Hawthorne Studies : Conducted between 1924 and 1932, the Hawthorne Studies were a series of case studies conducted by Elton Mayo and his colleagues to examine the impact of work environment on employee productivity. The studies were conducted at the Hawthorne Works plant of the Western Electric Company in Chicago and included interviews, observations, and experiments.
• The Stanford Prison Experiment: Conducted in 1971, the Stanford Prison Experiment was a case study conducted by Philip Zimbardo to examine the psychological effects of power and authority. The study involved simulating a prison environment and assigning participants to the role of guards or prisoners. The study was controversial due to the ethical issues it raised.
• The Challenger Disaster: The Challenger Disaster was a case study conducted to examine the causes of the Space Shuttle Challenger explosion in 1986. The study included interviews, observations, and analysis of data to identify the technical, organizational, and cultural factors that contributed to the disaster.
• The Enron Scandal: The Enron Scandal was a case study conducted to examine the causes of the Enron Corporation’s bankruptcy in 2001. The study included interviews, analysis of financial data, and review of documents to identify the accounting practices, corporate culture, and ethical issues that led to the company’s downfall.
• The Fukushima Nuclear Disaster : The Fukushima Nuclear Disaster was a case study conducted to examine the causes of the nuclear accident that occurred at the Fukushima Daiichi Nuclear Power Plant in Japan in 2011. The study included interviews, analysis of data, and review of documents to identify the technical, organizational, and cultural factors that contributed to the disaster.

Application of Case Study

Case studies have a wide range of applications across various fields and industries. Here are some examples:

Business and Management

Case studies are widely used in business and management to examine real-life situations and develop problem-solving skills. Case studies can help students and professionals to develop a deep understanding of business concepts, theories, and best practices.

Case studies are used in healthcare to examine patient care, treatment options, and outcomes. Case studies can help healthcare professionals to develop critical thinking skills, diagnose complex medical conditions, and develop effective treatment plans.

Case studies are used in education to examine teaching and learning practices. Case studies can help educators to develop effective teaching strategies, evaluate student progress, and identify areas for improvement.

Social Sciences

Case studies are widely used in social sciences to examine human behavior, social phenomena, and cultural practices. Case studies can help researchers to develop theories, test hypotheses, and gain insights into complex social issues.

Law and Ethics

Case studies are used in law and ethics to examine legal and ethical dilemmas. Case studies can help lawyers, policymakers, and ethical professionals to develop critical thinking skills, analyze complex cases, and make informed decisions.

Purpose of Case Study

The purpose of a case study is to provide a detailed analysis of a specific phenomenon, issue, or problem in its real-life context. A case study is a qualitative research method that involves the in-depth exploration and analysis of a particular case, which can be an individual, group, organization, event, or community.

The primary purpose of a case study is to generate a comprehensive and nuanced understanding of the case, including its history, context, and dynamics. Case studies can help researchers to identify and examine the underlying factors, processes, and mechanisms that contribute to the case and its outcomes. This can help to develop a more accurate and detailed understanding of the case, which can inform future research, practice, or policy.

Case studies can also serve other purposes, including:

• Illustrating a theory or concept: Case studies can be used to illustrate and explain theoretical concepts and frameworks, providing concrete examples of how they can be applied in real-life situations.
• Developing hypotheses: Case studies can help to generate hypotheses about the causal relationships between different factors and outcomes, which can be tested through further research.
• Providing insight into complex issues: Case studies can provide insights into complex and multifaceted issues, which may be difficult to understand through other research methods.
• Informing practice or policy: Case studies can be used to inform practice or policy by identifying best practices, lessons learned, or areas for improvement.

Advantages of Case Study Research

There are several advantages of case study research, including:

• In-depth exploration: Case study research allows for a detailed exploration and analysis of a specific phenomenon, issue, or problem in its real-life context. This can provide a comprehensive understanding of the case and its dynamics, which may not be possible through other research methods.
• Rich data: Case study research can generate rich and detailed data, including qualitative data such as interviews, observations, and documents. This can provide a nuanced understanding of the case and its complexity.
• Holistic perspective: Case study research allows for a holistic perspective of the case, taking into account the various factors, processes, and mechanisms that contribute to the case and its outcomes. This can help to develop a more accurate and comprehensive understanding of the case.
• Theory development: Case study research can help to develop and refine theories and concepts by providing empirical evidence and concrete examples of how they can be applied in real-life situations.
• Practical application: Case study research can inform practice or policy by identifying best practices, lessons learned, or areas for improvement.
• Contextualization: Case study research takes into account the specific context in which the case is situated, which can help to understand how the case is influenced by the social, cultural, and historical factors of its environment.

Limitations of Case Study Research

There are several limitations of case study research, including:

• Limited generalizability : Case studies are typically focused on a single case or a small number of cases, which limits the generalizability of the findings. The unique characteristics of the case may not be applicable to other contexts or populations, which may limit the external validity of the research.
• Biased sampling: Case studies may rely on purposive or convenience sampling, which can introduce bias into the sample selection process. This may limit the representativeness of the sample and the generalizability of the findings.
• Subjectivity: Case studies rely on the interpretation of the researcher, which can introduce subjectivity into the analysis. The researcher’s own biases, assumptions, and perspectives may influence the findings, which may limit the objectivity of the research.
• Limited control: Case studies are typically conducted in naturalistic settings, which limits the control that the researcher has over the environment and the variables being studied. This may limit the ability to establish causal relationships between variables.
• Time-consuming: Case studies can be time-consuming to conduct, as they typically involve a detailed exploration and analysis of a specific case. This may limit the feasibility of conducting multiple case studies or conducting case studies in a timely manner.
• Resource-intensive: Case studies may require significant resources, including time, funding, and expertise. This may limit the ability of researchers to conduct case studies in resource-constrained settings.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Questionnaire – Definition, Types, and Examples

Observational Research – Methods and Guide

Quantitative Research – Methods, Types and...

Qualitative Research Methods

Explanatory Research – Types, Methods, Guide

Survey Research – Types, Methods, Examples

Case studies

Case studies allow you to apply what you are learning to a real-world example, or ‘case’. The case could be either an actual or hypothetical situation, person, business or organisation.

Case studies usually ask you to:

• describe the case
• identify key issues of the case
• analyse the case using relevant theory or concepts
• make recommendations for the case.

General tips for case studies

Analyse the task carefully.

Case studies vary between disciplines, so read the assignment instructions and marking rubric carefully to understand what is expected.

Some case study assignments will ask you to answer very specific questions related to the case. For other case studies, you will decide on the content by interpreting the case information and connecting it with what you have been learning.

Read the case information thoroughly

Read the case information multiple times. You can use a range of reading strategies, including:

• skimming to get an idea of the case study scenario
• scanning to find specific information relevant to the assignment
• reading in detail to develop an in-depth understanding of the case.

While reading, keep the task in mind and ask yourself relevant questions, such as:

• How does this case relate to what I have been learning in class?
• What issues are evident in the case?
• What are the causes of these issues?
• What are some possible solutions to these issues?

Highlight any important information and make notes with your answers to these questions.

Connect the case, theory or concepts, and your analysis

Use a mind map or table to help you make connections between the case and theory or concepts. Draw on what you remember from classes and prescribed readings, and then conduct further research to help you analyse the case.

Use a mind map or table to help you plan the structure and content of your writing.

Example mindmap - Nursing

Example table for note-taking - Cybersecurity

This table shows how the student has analysed case evidence to identify problems in the case. The student has then researched relevant theory and made recommendations to solve the problems.

Writing a case study

Case studies are commonly written in a report  or essay format. Regardless of the format, all case studies should include three elements:

• facts from the case
• theory or concepts from literature (e.g. books, journals)
• your analysis.

This paragraph includes:

• case evidence (about Darren's drinking and smoking)
• theory (about the effect of alcohol and smoking and pain) and in-text citations (Chiang et al., 2016; Miyoshi, 2007)
• analysis (the implications for Darren's nursing care after the operation) 

Darren’s potential for pain is also influenced by lifestyle factors including excessive drinking and long-term smoking. [Case evidence] Darren identifies as a social drinker and consumes two beers each day after work and up to ten beers per day on weekends. [Theory] Frequent consumption of alcohol at high levels can increase a postoperative patient’s need for pain-relieving opioid medications, due to stimulating neuropathic pain delivered through both peripheral nervous system and central nervous system (CNS) pathways (Miyoshi, 2007, p. 208). [Case evidence] In addition to his high alcohol consumption, Darren is a current smoker and has smoked 30 cigarettes per day for the past 20 years. [Theory] Chiang et al. (2016) found that patients who smoke metabolise analgesic medications faster than non-smokers, as nicotine stimulates the CNS and produces analgesia in low quantities, resulting in smokers having a lower pain tolerance or hyperalgesia. [Analysis] Therefore, Darren’s alcohol use and smoking status mean that he is likely to experience higher levels of pain and require higher amounts of analgesia following his operation. 

Paragraph adapted and used with student permission

• case evidence (about access to Green Coin Company's network)
• theory (about attacks on passwords and Windows XP) and in-text citations (CVE 2014; Kapersky 2007; Windows 2017)
• analysis (the consequences of an attack on the company)

A major ICT weakness in Green Coin Company is that [Case evidence] employees access the network through a Windows XP frontend with a password authentication system. Passwords can be an effective way to authenticate into the system, provided that the password is strong and that the user keeps it secret. [Theory] However, a common attack vector on passwords is a trial-and-error approach where the attacker tries passwords repeatedly until they guess correctly and gain access to the network (Kapersky 2019). On a system with good security controls, the account will be locked out for a specified period after several wrong guesses, and an admin will be alerted that the account has been locked out (Windows 2017). However, Windows XP has a vulnerability that can bypass the lockout system, giving an attacker the ability to guess the password as many times as needed without being detected by the system (CVE 2014). [Analysis] If the attacker compromises an account that has access to the database, they will be able to add fake sales records, shipping requests and bullion transactions, thus compromising the integrity of the overall database. 

For more complete example case studies with further annotations, see the Word and PDF documents below.

Example case studies

• Example case study - Nursing [Word - 68KB]
• Example case study - Nursing [PDF - 170KB]
• Example case study - Cybersecurity [Word - 61KB]
• Example case study - Cybersecurity [PDF - 168KB]

Pathfinder link

Still have questions? Do you want to talk to an expert? Peer Learning Advisors or Academic Skills and Language Advisors  are available.

• << Previous: Reports
• Next: Reflective writing >>

How to write a case study — examples, templates, and tools

It’s a marketer’s job to communicate the effectiveness of a product or service to potential and current customers to convince them to buy and keep business moving. One of the best methods for doing this is to share success stories that are relatable to prospects and customers based on their pain points, experiences, and overall needs.

That’s where case studies come in. Case studies are an essential part of a content marketing plan. These in-depth stories of customer experiences are some of the most effective at demonstrating the value of a product or service. Yet many marketers don’t use them, whether because of their regimented formats or the process of customer involvement and approval.

A case study is a powerful tool for showcasing your hard work and the success your customer achieved. But writing a great case study can be difficult if you’ve never done it before or if it’s been a while. This guide will show you how to write an effective case study and provide real-world examples and templates that will keep readers engaged and support your business.

In this article, you’ll learn:

What is a case study?

How to write a case study, case study templates, case study examples, case study tools.

A case study is the detailed story of a customer’s experience with a product or service that demonstrates their success and often includes measurable outcomes. Case studies are used in a range of fields and for various reasons, from business to academic research. They’re especially impactful in marketing as brands work to convince and convert consumers with relatable, real-world stories of actual customer experiences.

The best case studies tell the story of a customer’s success, including the steps they took, the results they achieved, and the support they received from a brand along the way. To write a great case study, you need to:

• Celebrate the customer and make them — not a product or service — the star of the story.
• Craft the story with specific audiences or target segments in mind so that the story of one customer will be viewed as relatable and actionable for another customer.
• Write copy that is easy to read and engaging so that readers will gain the insights and messages intended.
• Follow a standardized format that includes all of the essentials a potential customer would find interesting and useful.
• Support all of the claims for success made in the story with data in the forms of hard numbers and customer statements.

Case studies are a type of review but more in depth, aiming to show — rather than just tell — the positive experiences that customers have with a brand. Notably, 89% of consumers read reviews before deciding to buy, and 79% view case study content as part of their purchasing process. When it comes to B2B sales, 52% of buyers rank case studies as an important part of their evaluation process.

Telling a brand story through the experience of a tried-and-true customer matters. The story is relatable to potential new customers as they imagine themselves in the shoes of the company or individual featured in the case study. Showcasing previous customers can help new ones see themselves engaging with your brand in the ways that are most meaningful to them.

Besides sharing the perspective of another customer, case studies stand out from other content marketing forms because they are based on evidence. Whether pulling from client testimonials or data-driven results, case studies tend to have more impact on new business because the story contains information that is both objective (data) and subjective (customer experience) — and the brand doesn’t sound too self-promotional.

Case studies are unique in that there’s a fairly standardized format for telling a customer’s story. But that doesn’t mean there isn’t room for creativity. It’s all about making sure that teams are clear on the goals for the case study — along with strategies for supporting content and channels — and understanding how the story fits within the framework of the company’s overall marketing goals.

Here are the basic steps to writing a good case study.

1. Identify your goal

Start by defining exactly who your case study will be designed to help. Case studies are about specific instances where a company works with a customer to achieve a goal. Identify which customers are likely to have these goals, as well as other needs the story should cover to appeal to them.

The answer is often found in one of the buyer personas that have been constructed as part of your larger marketing strategy. This can include anything from new leads generated by the marketing team to long-term customers that are being pressed for cross-sell opportunities. In all of these cases, demonstrating value through a relatable customer success story can be part of the solution to conversion.

2. Choose your client or subject

Who you highlight matters. Case studies tie brands together that might otherwise not cross paths. A writer will want to ensure that the highlighted customer aligns with their own company’s brand identity and offerings. Look for a customer with positive name recognition who has had great success with a product or service and is willing to be an advocate.

The client should also match up with the identified target audience. Whichever company or individual is selected should be a reflection of other potential customers who can see themselves in similar circumstances, having the same problems and possible solutions.

Some of the most compelling case studies feature customers who:

• Switch from one product or service to another while naming competitors that missed the mark.
• Experience measurable results that are relatable to others in a specific industry.
• Represent well-known brands and recognizable names that are likely to compel action.
• Advocate for a product or service as a champion and are well-versed in its advantages.

Whoever or whatever customer is selected, marketers must ensure they have the permission of the company involved before getting started. Some brands have strict review and approval procedures for any official marketing or promotional materials that include their name. Acquiring those approvals in advance will prevent any miscommunication or wasted effort if there is an issue with their legal or compliance teams.

3. Conduct research and compile data

Substantiating the claims made in a case study — either by the marketing team or customers themselves — adds validity to the story. To do this, include data and feedback from the client that defines what success looks like. This can be anything from demonstrating return on investment (ROI) to a specific metric the customer was striving to improve. Case studies should prove how an outcome was achieved and show tangible results that indicate to the customer that your solution is the right one.

This step could also include customer interviews. Make sure that the people being interviewed are key stakeholders in the purchase decision or deployment and use of the product or service that is being highlighted. Content writers should work off a set list of questions prepared in advance. It can be helpful to share these with the interviewees beforehand so they have time to consider and craft their responses. One of the best interview tactics to keep in mind is to ask questions where yes and no are not natural answers. This way, your subject will provide more open-ended responses that produce more meaningful content.

4. Choose the right format

There are a number of different ways to format a case study. Depending on what you hope to achieve, one style will be better than another. However, there are some common elements to include, such as:

• An engaging headline
• A subject and customer introduction
• The unique challenge or challenges the customer faced
• The solution the customer used to solve the problem
• The results achieved
• Data and statistics to back up claims of success
• A strong call to action (CTA) to engage with the vendor

It’s also important to note that while case studies are traditionally written as stories, they don’t have to be in a written format. Some companies choose to get more creative with their case studies and produce multimedia content, depending on their audience and objectives. Case study formats can include traditional print stories, interactive web or social content, data-heavy infographics, professionally shot videos, podcasts, and more.

5. Write your case study

We’ll go into more detail later about how exactly to write a case study, including templates and examples. Generally speaking, though, there are a few things to keep in mind when writing your case study.

• Be clear and concise. Readers want to get to the point of the story quickly and easily, and they’ll be looking to see themselves reflected in the story right from the start.
• Provide a big picture. Always make sure to explain who the client is, their goals, and how they achieved success in a short introduction to engage the reader.
• Construct a clear narrative. Stick to the story from the perspective of the customer and what they needed to solve instead of just listing product features or benefits.
• Leverage graphics. Incorporating infographics, charts, and sidebars can be a more engaging and eye-catching way to share key statistics and data in readable ways.
• Offer the right amount of detail. Most case studies are one or two pages with clear sections that a reader can skim to find the information most important to them.
• Include data to support claims. Show real results — both facts and figures and customer quotes — to demonstrate credibility and prove the solution works.

6. Promote your story

Marketers have a number of options for distribution of a freshly minted case study. Many brands choose to publish case studies on their website and post them on social media. This can help support SEO and organic content strategies while also boosting company credibility and trust as visitors see that other businesses have used the product or service.

Marketers are always looking for quality content they can use for lead generation. Consider offering a case study as gated content behind a form on a landing page or as an offer in an email message. One great way to do this is to summarize the content and tease the full story available for download after the user takes an action.

Sales teams can also leverage case studies, so be sure they are aware that the assets exist once they’re published. Especially when it comes to larger B2B sales, companies often ask for examples of similar customer challenges that have been solved.

Now that you’ve learned a bit about case studies and what they should include, you may be wondering how to start creating great customer story content. Here are a couple of templates you can use to structure your case study.

Template 1 — Challenge-solution-result format

• Start with an engaging title. This should be fewer than 70 characters long for SEO best practices. One of the best ways to approach the title is to include the customer’s name and a hint at the challenge they overcame in the end.
• Create an introduction. Lead with an explanation as to who the customer is, the need they had, and the opportunity they found with a specific product or solution. Writers can also suggest the success the customer experienced with the solution they chose.
• Present the challenge. This should be several paragraphs long and explain the problem the customer faced and the issues they were trying to solve. Details should tie into the company’s products and services naturally. This section needs to be the most relatable to the reader so they can picture themselves in a similar situation.
• Share the solution. Explain which product or service offered was the ideal fit for the customer and why. Feel free to delve into their experience setting up, purchasing, and onboarding the solution.
• Explain the results. Demonstrate the impact of the solution they chose by backing up their positive experience with data. Fill in with customer quotes and tangible, measurable results that show the effect of their choice.
• Ask for action. Include a CTA at the end of the case study that invites readers to reach out for more information, try a demo, or learn more — to nurture them further in the marketing pipeline. What you ask of the reader should tie directly into the goals that were established for the case study in the first place.

Template 2 — Data-driven format

• Start with an engaging title. Be sure to include a statistic or data point in the first 70 characters. Again, it’s best to include the customer’s name as part of the title.
• Create an overview. Share the customer’s background and a short version of the challenge they faced. Present the reason a particular product or service was chosen, and feel free to include quotes from the customer about their selection process.
• Present data point 1. Isolate the first metric that the customer used to define success and explain how the product or solution helped to achieve this goal. Provide data points and quotes to substantiate the claim that success was achieved.
• Present data point 2. Isolate the second metric that the customer used to define success and explain what the product or solution did to achieve this goal. Provide data points and quotes to substantiate the claim that success was achieved.
• Present data point 3. Isolate the final metric that the customer used to define success and explain what the product or solution did to achieve this goal. Provide data points and quotes to substantiate the claim that success was achieved.
• Summarize the results. Reiterate the fact that the customer was able to achieve success thanks to a specific product or service. Include quotes and statements that reflect customer satisfaction and suggest they plan to continue using the solution.
• Ask for action. Include a CTA at the end of the case study that asks readers to reach out for more information, try a demo, or learn more — to further nurture them in the marketing pipeline. Again, remember that this is where marketers can look to convert their content into action with the customer.

While templates are helpful, seeing a case study in action can also be a great way to learn. Here are some examples of how Adobe customers have experienced success.

Juniper Networks

One example is the Adobe and Juniper Networks case study , which puts the reader in the customer’s shoes. The beginning of the story quickly orients the reader so that they know exactly who the article is about and what they were trying to achieve. Solutions are outlined in a way that shows Adobe Experience Manager is the best choice and a natural fit for the customer. Along the way, quotes from the client are incorporated to help add validity to the statements. The results in the case study are conveyed with clear evidence of scale and volume using tangible data.

The story of Lenovo’s journey with Adobe is one that spans years of planning, implementation, and rollout. The Lenovo case study does a great job of consolidating all of this into a relatable journey that other enterprise organizations can see themselves taking, despite the project size. This case study also features descriptive headers and compelling visual elements that engage the reader and strengthen the content.

Tata Consulting

When it comes to using data to show customer results, this case study does an excellent job of conveying details and numbers in an easy-to-digest manner. Bullet points at the start break up the content while also helping the reader understand exactly what the case study will be about. Tata Consulting used Adobe to deliver elevated, engaging content experiences for a large telecommunications client of its own — an objective that’s relatable for a lot of companies.

Case studies are a vital tool for any marketing team as they enable you to demonstrate the value of your company’s products and services to others. They help marketers do their job and add credibility to a brand trying to promote its solutions by using the experiences and stories of real customers.

When you’re ready to get started with a case study:

• Think about a few goals you’d like to accomplish with your content.
• Make a list of successful clients that would be strong candidates for a case study.
• Reach out to the client to get their approval and conduct an interview.
• Gather the data to present an engaging and effective customer story.

Adobe can help

There are several Adobe products that can help you craft compelling case studies. Adobe Experience Platform helps you collect data and deliver great customer experiences across every channel. Once you’ve created your case studies, Experience Platform will help you deliver the right information to the right customer at the right time for maximum impact.

To learn more, watch the Adobe Experience Platform story .

Keep in mind that the best case studies are backed by data. That’s where Adobe Real-Time Customer Data Platform and Adobe Analytics come into play. With Real-Time CDP, you can gather the data you need to build a great case study and target specific customers to deliver the content to the right audience at the perfect moment.

Watch the Real-Time CDP overview video to learn more.

Finally, Adobe Analytics turns real-time data into real-time insights. It helps your business collect and synthesize data from multiple platforms to make more informed decisions and create the best case study possible.

Request a demo to learn more about Adobe Analytics.

https://business.adobe.com/blog/perspectives/b2b-ecommerce-10-case-studies-inspire-you

https://business.adobe.com/blog/basics/business-case

https://business.adobe.com/blog/basics/what-is-real-time-analytics

• Numerical Reasoning
• Verbal Reasoning
• Inductive Reasoning
• Diagrammatic Reasoning
• Logical Reasoning
• Mechanical Reasoning
• Situational Judgement
• Deductive reasoning
• Critical thinking
• Spatial reasoning
• Error checking
• Verbal comprehension
• Reading comprehension
• Psychometric tests
• Personality test
• In-Tray exercise
• E-Tray exercise
• Group exercise
• Roleplay exercise
• Presentation exercise
• Analysis exercise
• Case study exercise
• Game based assessments
• Competency based assessment
• Strengths based assessment
• Strengths based interview
• Video interview
• Saville Assessment
• Talent Q / Korn Ferry
• Watson Glaser
• Test Partnership
• Clevry (Criterion)
• Criteria Corp
• Aon / Cut-e
• Sova Assessment
• For Practice
• For Business
• Case Study Exercise

Case Study Exercises are commonly used in assessment centres, and often are unique to each company.

• Buy Full Practice Tests
• What is a Case Study exercise

Page contents:

How do case study exercises work.

Updated: 08 September 2022

Assessment Centre Exercises:

• Analysis Exercise
• Role Play Exercise
• Group Exercise
• Presentation Exercise

During an assessment day, it is common that you need to undertake a case study exercise . These exercises place candidates in real-life situations where they are tasked with solving problems faced by professionals in the real world. A case study typically involves being given various documents containing different information, either detailing a problem or situation that needs dealing with and requiring the candidate to resolve the issue at hand by formulating a plan. The problems or situation in the case study will be similar if not identical to problems encountered in the role itself. Candidates are also provided with background information to the elements of the case study, whether these be details of fictitious companies or sales figures, or other. The resolutions or solutions provided by the candidate regarding the problems are part of the assessment centre performance rating.

Why are case study exercises used?

Case study exercises are proficient predictors of role performance as they will resemble the work being done on the job. Therefore, case study exercises typically tilt highly on an assessment centre rating for candidates. Likewise, if a presentation exercise is required after the case study, based on details brought up during the case study, then your case study rating will likely impact your presentation exercise rating. Equally, this may manifest into the role play exercise which will do a similar thing to the presentation exercise – carrying on the case study situation. It is also entirely possible for the case study to be continued in a group exercise – which evaluate a candidate’s ability to work in a team. Given all this, you will need to perform well in the case study exercise to ensure a high rating.

What will the case study exercise be like?

As mentioned, the case study exercise you will be asked to perform will be similar to the type of work you will have to do in the role you are applying for.

The case study exercise may be purchased off the self from a test provider who specialize in the test style. This will mean that it won't be fully specific to the company you are applying to, but will be related to the role. Likewise, it can be designed bespoke if the organization requires specific role assessment. It's likely the larger and harder to get into the company is, the more tailored their exercises will be.

How can I prepare for the case study exercise?

Analysing technical documents and company reports may be helpful practice in preparation for a case study exercise. This will give a chance to familiarize yourself with the types of information typically found in these documents, and thus the case study exercise. Practicing case study exercises will also act as great preparation and they will provide a great insight into how they work and how they are to be handled. This will also prevent any unnecessary unknowns you could have before taking a case study exercise, as you will have already experienced how they work in practice.

We have an assessment centre pack which contains an example of the exercises you could face.

• AON Hewitt G.A.T.E.
• PI Cognitive Assessment (PLI Test)
• Korn Ferry Leadership Assessment
• Berke Assessment
• Ergometrics
• Thomas International
• Predictive Index (PI)
• NEO Personality Inventory
• Leadership Assessment
• Gallup’s CliftonStrengths
• Sales Personality Tests
• Personality Management Tests
• Saville Wave
• McQuaig Word Survey
• Bell Personality Test
• Myers Briggs Personality Test
• DISC Personality Test
• Management SJT
• Supervisory SJT
• Administrative SJT
• Call Center SJT
• Customer Service SJT
• Firefighter SJT
• Numerical Reasoning Tests
• Verbal Reasoning Tests
• Logical Reasoning Tests
• Cognitive Ability Tests
• Technical Aptitude Tests
• Spatial Reasoning Tests
• Abstract Reasoning Test
• Deductive Reasoning Tests
• Inductive Reasoning Tests
• Mechanical Reasoning Tests
• Diagrammatic Reasoning Tests
• Fault Finding Aptitude Tests
• Mathematical Reasoning Tests
• Critical Thinking Tests
• Analytical Reasoning Tests
• Raven’s Progressive Matrices Test
• Criteria’s CCAT
• Matrigma Test
• Air Traffic Controller Test
• Administrative Assistant Exam
• Clerical Ability Exam
• School Secretary Tests
• State Trooper Exam
• Probation Officer Exam
• FBI Entrance Exam
• Office Assistant Exam
• Clerk Typist Test
• Police Records Clerk Exam
• Canada’s Public Service Exams
• Firefighter Exams
• Police Exams
• Army Aptitude Tests
• USPS Postal Exams
• Hiring Process by Professions

Select Page

Assessment Center Case Studies Practice & Tips – 2024

Case studies are a central part of the exercises making up most assessment centers . Employers use them to provide valuable insight into the applicants. They provide a way to assess a graduate or job-seeker’s capability and their potential performance after selection. To do this, the assessment center presents the candidate with a simulated situation that might be faced on the actual job and waits to see how the candidate will respond. The information assessors collect proves invaluable to companies as they work through the screening and hiring process with the candidates who are most likely to perform well in the job opening.

What Is a Case Study Exercise?

Case studies are simulation exercises that put a candidate into situations they might actually see while on the job. The exercises can be done as a group or individually. Which it is will depend on the employer and the assessment center. The case studies typically provide information that includes financial reports, market studies, or competition analysis and other information that may relate to any aspect of the profession. It may also provide other company reports, consultant’s reports, new product research results, and more. This makes the exercise similar in some ways to an in-tray exercise though the documents are longer for a case study.

Key Features of Case Studies

The exercise can be presented at the end either in written report format or as a presentation, depending on the preference of those running the exam. The assessors then evaluate the candidate’s ability to analyze information with a logical approach to decision making and their aptitude for tackling difficult situations. From there, they score performance.

Case study exercises often are based on a few core topics. Some of these include:

• Finding the feasibility and profitability for the introduction of a new product or service
• Merger, acquisition, or joint venture related managerial decisions
• Annual report evaluation and profitability and loss analysis
• Task prioritization and problem-solving with a given deadline

Many times, the case study’s theme or scenario provides the stage for other assessment center exercises, so paying attention to what the scenario is and the information provided about it can prove helpful in further exercises. If this is the case, the problem-solving case study is likely to show up as one of the first few exercises you do after re-taking the necessary psychometric aptitude assessments for score confirmation.

Competencies Required for Case Studies

The key competencies that case study exercises usually assess are:

• Analytical thinking and assimilation of information
• Commercial awareness and Innovation
• Organization
• Decisiveness and Judgment

The goal of the exercise is to review and analyze the given information to come up with solid business decisions. The assessors will look at both the decision reached and the logical justification for the recommendations. Because of this, the test is not designed to have one ‘correct’ answer. Instead, it is concerned with the approach to solving the issue as much as it is with the solution.

This is the point in the assessment and pre-hiring process where candidates should show the recruiters what they can do. Usually, the exercise lasts around forty minutes. Employers may use either fictional examples or, in some cases, even real live projects with the sensitive information replaced for fictional information.

Due to the nature of the exercise, job-seekers and graduates taking this type of assessment should possess several key skills. They must be able to interpret large quantities of data from multiple sources and in varying formats, use analytical and strategic analysis to solve problems, formulate and commit to a decision, demonstrate commercial and entrepreneurial insight on a problem, and use oral communication skills to discuss the decisions made and the reasoning behind them. Without these key abilities, case exercises may prove challenging for individuals.

How to Prepare for Case Study Exercises?

With the large amount of information presented on assessment center case studies and the many things to consider, it can be difficult to know where to start. Particularly for those participating in a graduate assessment center case studies with no prior experience with assessment centers, the case study may seem daunting.

However, it is possible to prepare with some case study practice and by reviewing assessment case study examples similar to the ones that will be given in your assessment center. These tips for preparation and practice as well the day of will help those facing a case study assessment to do so with confidence.

Case Studies: Tips for Success

Review the advice below as you begin to prepare for the assessment center:

• If it is a group exercise , show the recruiters you can work with the team.
• For a group exercise, determine what roles individuals in the scenario are associated with and how they may interact with your or impact the analysis and decision-making process.
• Determine what information needs to be kept and what should be discarded as early on as possible.
• Manage time carefully and plan your approach based on the time available to you.
• Consider all possible solutions and analyze them carefully before choosing a decision.
• When finished, ensure that you have a solid foundation for the proposal and a plan of action to implement for your chosen solution.
• Make sure you communicate that foundation and the logic behind your decision.
• When presenting as a group, actively participate but avoid dominating the conversation or situation.
• Gather information on the organization, job profile, and any other data that could be in the case study to be prepared before assessment center day if possible.
• If you do not need to present for a group exercise, consider nominating yourself as someone who can respond to questions.
• Practice structuring and delivering presentations in a case study format before testing.

If you follow the advice above and put in enough time practicing and preparing to feel confident, you should be able to ace this portion of your assessment center. Remember that the solution is not the most important thing about this exercise. How you work with others and the reasoning behind your answer is. So, use the time you have wisely and do not overlook anything as you work to come to a good solution. As you do this, relax and use this as a chance to show the recruiters that you really know what you said you did during the interview stage . That is what this exam is about.

Assessment Center

• Written Exercises
• Job Interviews
• Competency-Based Interview Q&A
• In-Tray Exercise
• Case Studies
• Group Exercises

Related Links:

• Situational Judgement Tests (SJT)
• Job Personality Tests
• Aptitude Tests
• Civil Service Exams
• Police Officer Exams
• Firefighters Exams
• Assessment Companies
• Assessment Centers Guide – 2024
• Hiring Companies – Find an Employer

National Institute for Learning Outcomes Assessment

• Mission & Vision
• National Advisory Panel
• NILOA Staff
• Partner & Collaborating Organizations
• Occasional Papers
• Assessment in Practice
• Case Studies
• Assignment Library
• Featured Assignments
• Assignment Charrette
• Equity Responses
• Transparency Framework
• Excellence in Assessment
• Evidence-Based Storytelling
• Featured Websites Archive
• Curriculum Mapping
• Learning Recognition Collaborative
• Degree Qualifications Profile
• Assessment Resources
• Newsletters
• NILOA Surveys Archive
• Resources for Newcomers
• LARC Assessment Modules
• Questions at Hand
• Audience Type

Case studies provide short, instructive examples of good assessment practice, along with lessons learned that are widely applicable.

Alverno College is a case study site due to innovative and long-standing assessment practices as well as a commitment to student-centered teaching. Although Alverno is well established in its assessment practices, other institutions can learn from and adapt many aspects of Alverno’s assessment model to improve and sustain their own meaningful assessment work.

American Public University System (APUS) is a case study site for its unique mission and significant headway made in working with the Degree Qualifications Profile (DQP), leading to institution-wide implementation and adoption, the thoughtful process for working through DQP use, the development of signature assignments, and the incorporation of the DQP framework into the university system’s program review process.

Augustana College is a case study site due to its approach to faculty involvement and a long-standing commitment to assessing and communicating student learning. Augustana’s focus on teaching and learning, the dynamic role of the Assessment Review Committee, and communication strategies has allowed them to make several improvements on campus based on their assessment activities.

Capella University is a case study site for its systematic, embedded student learning outcomes assessment process; its administrative support and vision of what assessment can do for individual learners; its transparency efforts such as Capella Results, which publicizes assessment results; and its use of assessment results to enhance learner success.

Carnegie Mellon University (CMU) was selected as a case study for the approach to student learning outcomes assessment that reflects the institution’s commitment to interdisciplinarity and innovative teaching and learning. Three elements have been instrumental in CMU’s advances in program-level student learning outcomes assessment: 1) an institutionalized research-oriented and data-informed university decision-making process driven by deans and departments; 2) an organizational culture with established processes promoting continuous improvement; and 3) the elevation of a cross-campus faculty resource–the Eberly Center for Teaching Excellence–as the hub of assessment support. The original case study took place in 2012 with a follow-up study published in 2021 .

Colorado State University publicized its commitment to ensuring transparency and accountability to students, parents, and the public; expanded its continuous improvement system for managing information sharing to serve the decision-making and reporting needs of various audiences through the CSU Plan for Researching Improvement and Supporting Mission, or PRISM; uses a peer review system for feedback, and serves as a model for bridging the work of academic affairs and student affairs together through student learning outcomes assessment.

Daemen College is a case study site due to their involvement in the Council of Independent Colleges (CIC) consortium on the Degree Qualifications Profile (DQP). As part of this project, Daemen College engaged with the DQP to further their preexisting curricular mapping projects, as well as advance their newly instituted assessment program and assignment design initiative, including a co-curricular inventory.

Georgia State University (GSU) and Georgia Perimeter College (GPC) were selected as case study sites for its work in testing the Degree Qualifications Profile (DQP) to facilitate transfer. The University of Georgia System project was a partnership between GSU and GPC to explore the application of the DQP to improve the success of transfer students in biology, psychology, and criminal justice programs that involve high numbers of transfer students between the two institutions.

Indiana University Purdue University Indianapolis (IUPUI) was invited to write a case study because of its strong and rich history of using numerous forms of applied and experiential learning to promote student engagement along with its ongoing Comprehensive Learner Record (CLR) work.

Juniata College was identified as a case study site for the faculty-led Center for the Scholarship of Teaching and Learning (SoTL Center) that champions and supports evidence-based teaching; an administration-supported accountability website that provides data and information about outcomes to multiple audiences; and the use of evidence of student learning to make improvements at the institution and individual course levels.

Kansas City Kansas Community College (KCKCC) is a case study site due to its creation of an alternative system for documenting student achievement of Degree Qualifications Profile (DQP) proficiencies. Based on an interactive curriculum mapping database where faculty enter information about individual student performance on each learning outcome and competency in their courses, reports are generated for students and programs to review and direct future action.

LaGuardia Community College is a case study site due to its reputation as a leader in learning outcomes assessment, particularly through the use of electronic portfolios (ePortfolios), commitment to assessment, collaboration across units at the college, and the institution’s robust program review system which includes assessment.

McKendree University is a case study site for its crosswalk of various learning frameworks (such as the Degree Qualifications Profile, LEAP Essential Learning Outcomes, National Collegiate Athletic Association’s (NCAA) Division II Life and Balance key attributes) to McKendree’s student learning outcomes, as well as the deliberate process of gaining campus awareness and support through their committee structure and learning outcome timeline. The original case study took place in 2016 with a follow-up study published in 2020 .

National Louis University   is a Degree Qualifications Profile (DQP) case study institution due to the intentional use of the DQP in the design of its Pathways Program including competency development, curricular focus, and assessment alignment. NLU’s use of the DQP provides an example for other institutions interested in using the DQP to develop new programs as well as curricular pathways that include a focus upon differentiated levels of learning.

North Carolina A&T State University is a case study site due to the commitment to improving the institution by developing a culture of inquiry through administrative leadership that encourages discussions and collaboration around student learning outcomes assessment activities; the use of professional development opportunities to help foster the involvement and commitment of faculty members; and the systematic and intentional use of student feedback.

Palo Alto College was selected as a NILOA case study based on its successful efforts in adapting NILOA’s assignment design toolkit to engage faculty, staff and students in assessment. Offering intimate workshops frequently throughout the academic calendar year has created a ground swell of faculty reinvesting themselves in the curriculum.

Point Loma Nazarene University is a case study site for its involvement of faculty and staff across a range of academic fields to move beyond conversations about outcomes and curriculum alignment to significant assessment activity that is comparable across programs, as well as their efforts in transparency of process and practice.

St. Olaf College is a case study institution due to the framing of assessment as inquiry in support of student learning that is meaningful, manageable, and mission-driven; the utilization-focus/backward-design approach employed in assessment; the integration of student learning outcomes assessment processes into faculty governance structures; along with the collaborative involvement of multiple stakeholders and diverse ways in which evidence of student learning is utilized throughout the institution. The original case took place in 2012 and the update occurred in 2020 .

Texas A&M International University (TAMIU) is a case study site for its commitment to choosing assessment measures and tools appropriate for its students; its long history with and innovative approach to assessment; and the influential role of professional development at the institution to help prepare “Assessment Champions and expand the number of pockets of excellence in terms of assessment practices throughout the campus.

University of Illinois at Urbana-Champaign’s case study provides insight into how the Department of African American Studies at UIUC utilizes assessment in course design, being mindful of aligning diversity and inclusion outcomes within the course and program goals, and ensuring students attain these Case Study outcomes in both in-person and online courses utilizing equitable assessments and student involvement in the assessment process

Utah State University is a case study site for the faculty-led involvement in the state of Utah Tuning projects; integration of the Degree Qualifications Profile (DQP) with various programs and colleges on campus; and bridging the work of national initiatives such as Utah’s status as a LEAP state, participation in AAC&U’s Quality Collaboratives project, involvement with the Multi-State Collaborative and WICHE Passports Initiative, general education revision; and its integration of High-Impact Practices to make connections across the entire institution to better serve students.

Washington State University (WSU) is a case study site because of its promising approach to student learning outcomes assessment in the often-challenging context of a large, highly decentralized research university, characterized by a deliberately incremental and iterative process, moving the institution step-by-step toward habits, practices, and policies that support ongoing educational improvement.

Privacy Overview

Virtual Patient: James- Assessment (SW)

Simulation Created on January 3, 2024

Cost: This simulation is included in a Simucase Membership .

Simulation Participation Information

This is an interactive Simucase simulation. The virtual patient James- Assessment (SW) will interact with you through your web browser over the internet. To work with James- Assessment (SW) you must have the following:

• A high-speed, broadband internet connection (wired preferred)
• HTML5 Browser with audio/video Chrome Recommended
• JavaScript enabled
• Cookies enabled

Please click here to test your browser before purchasing our Simucase Membership.

Visit our Contact us page or give us a call if you have questions.

Evolving Evidence-Based Value Assessment of One-Time Therapies: Tisagenlecleucel as a Case Study

• Original Research Article
• Open access
• Published: 29 April 2024

Cite this article

You have full access to this open access article

• Theodore Laetsch 1 ,
• Jie Zhang 2 ,
• Hongbo Yang 3 ,
• Yanwen Xie 4 ,
• Dudan Zhang 4 &
• Louis Garrison 5  

73 Accesses

Explore all metrics

Economic evaluation of one-time therapies during reimbursement decision-making is challenging due to uncertain long-term outcomes. The availability of 5-year outcome data from the ELIANA trial and real-world evidence of tisagenlecleucel, the first chimeric antigen receptor T-cell (CAR-T) therapy, presents an opportunity to re-evaluate the predictions of prior cost-effectiveness analyses (CEAs).

To conduct a systematic literature review (SLR) of prior CEAs of tisagenlecleucel for pediatric/young adult relapsed or refractory acute lymphoblastic leukemia (r/r ALL) and evaluate the impact of recently available 5-year efficacy data from ELIANA and advances in CAR-T manufacturing in an updated CEA model.

OVID MEDLINE/Embase and health technology assessment (HTA) databases were searched for full-text economic evaluations in English reporting cost-effectiveness results for tisagenlecleucel for r/r ALL. Evaluations with publicly reported incremental cost-effectiveness ratios (ICERs) were included in the SLR. Study screening and data abstraction were conducted following PRISMA guidelines . Data extracted included the country/currency, perspective, clinical trial evidence, model structures, long-term efficacy extrapolation approaches (i.e., overall survival [OS]), time horizon, discount rates, and outcomes (i.e., life years [LY], quality-adjusted LY [QALY], and ICERs). The CEA model reported in Wakase et al. was updated using 5-year OS data from ELIANA and the CAR-T infusion rate informed by real-world practice.

Sixteen records corresponding to 15 unique studies were included in the SLR (11 publications and 5 HTA reports); all were conducted from the health care system perspective of the respective countries. Most studies found tisagenlecleucel to be cost effective, but all studies’ projected 3- and 5-year OS rates for tisagenlecleucel were lower than the observed 3- and 5-year rates, respectively, derived from 5-year ELIANA data. When applying updated OS projections from the most recent ELIANA data cut and higher infusion rates of 92.5% (per the real-world infusion rate)—96.0% (per the manufacturer success rate) to the CEA of Wakase et al., the associated QALYs for tisagenlecleucel increased from 11.6 to 14.6–15.0, and LYs increased from 13.3 to 17.0–17.5. Accordingly, the ICERs for tisagenlecleucel decreased from ¥2,035,071 to ¥1,787,988–¥1,789,048 versus blinatumomab and from ¥2,644,702 to ¥2,257,837–¥2,275,181 versus clofarabine combination therapy in the updated CEA model.

Conclusions and Relevance

Projections at launch of the likely cost effectiveness of tisagenlecleucel appear to have underestimated its ultimate economic value given more recent trial and real-world data. To balance uncertainty in initial valuation with the need to provide access to novel oncology therapies, payers can consider flexible reimbursement policies alongside ongoing assessments as new data emerge.

Similar content being viewed by others

Cost-effectiveness Analysis of Tisagenlecleucel Versus Blinatumomab in Children and Young Adults with Acute Lymphoblastic Leukemia: Partitioned Survival Model to Assess the Impact of an Outcome-Based Payment Arrangement

Cost-Effectiveness of KTE-X19 for Adults with Relapsed/Refractory B-Cell Acute Lymphoblastic Leukemia in the United States

Tisagenlecleucel for the Treatment of Relapsed or Refractory B-cell Acute Lymphoblastic Leukaemia in People Aged up to 25 Years: An Evidence Review Group Perspective of a NICE Single Technology Appraisal

Avoid common mistakes on your manuscript.

1 Introduction

The past decade has brought remarkable innovation in cell and gene therapies aiming to provide long-term benefit through a single treatment. However, it is challenging to assess the value of novel, highly-active therapies at regulatory approval for reimbursement decision-making due to limited long-term data and frequent lack or infeasibility of head-to-head trials comparing them with standard of care (SOC) options [ 1 ]. If early data indicate promising long-term clinical benefit and curative potential for a disease with high unmet need, it creates urgency to develop solutions for timely access. In addition to accumulation of long-term data, several factors occurring after approval can impact cost effectiveness. The cellular therapy manufacturing and delivery processes are novel and continuously evolving, demonstrated by higher infusion success rates and shorter waiting times in the commercial setting than those observed in early clinical trials [ 2 , 3 ]. Over time, the medical community’s accumulated experience contributes to the optimal utilization of these therapies, often among a population broader than that in clinical trials, and improved best practices for managing potential adverse events (AEs). Often, new and expanded indications pose additional challenges for cell and gene therapy reimbursement decisions.

These features are evidenced in emerging data from tisagenlecleucel, the first chimeric antigen receptor T cell (CAR-T) therapy made available to patients when it was approved in the USA for B-cell precursor acute lymphoblastic leukemia (ALL) in children and young adults (aged up to 25 years) who were in their second or later relapse [ 4 , 5 ]. B-cell ALL is one of the most common malignancies diagnosed in children and approximately 15% of pediatric and young adults with ALL ultimately relapse or experience refractory (r/r) disease after first-line treatment, which is associated with increased morbidity and mortality [ 6 , 7 , 8 ]. The approval of tisagenlecleucel was based on the pivotal single-arm, Phase II ELIANA trial, wherein 81% of infused patients achieved remission, accompanied by meaningful quality-of-life gains [ 9 ]. In contrast to prior SOC therapies for r/r ALL such as salvage chemotherapy (SC), tisagenlecleucel targets the tumor with high precision and can therefore be effective for aggressive disease where other therapies have failed [ 10 ].

Since the first approval of tisagenlecleucel in 2017, the success rate of manufacturing patients’ autologous CAR-T cells has improved, waiting times for infusion have decreased, and updates to the management of cytokine release syndrome and neurotoxicity have resulted in better tolerance and lower toxicity [ 11 , 12 , 13 ]. As it has been more than 5 years since the initial approval of tisagenlecleucel, we now have access to its related real-world and long-term trial data. Thus, there is an opportunity to evaluate how a one-time therapy, such as tisagenlecleucel, was initially evaluated by health technology assessment (HTA) agencies and in cost-effectiveness analyses (CEAs), to understand whether their assumptions were realistic and to determine whether modifications to existing approaches for assessing the value of one-time therapies are warranted.

Accordingly, we aimed to evaluate the impact of long-term data from ELIANA and evidence accumulated from real-world clinical practice on the economic value of tisagenlecleucel. First, we conducted a systematic literature review (SLR) of CEAs of tisagenlecleucel for ALL and summarized the clinical trial evidence used, long-term efficacy extrapolation approaches, and conclusions. Next, we conducted analyses incorporating long-term, 5-year efficacy data from ELIANA (data cutoff: November 17, 2022; median follow-up: 2.2 years; data on file, Novartis), as well as real-world data on improvements in CAR-T infusion rate, to understand their impact on the outcomes of a recent CEA of tisagenlecleucel for r/r ALL [ 14 ].

2.1 Systematic Literature Review

The methods for performing the SLR followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Fig. 1 ; refer to Supplemental Methods for the search strategy). A systematic literature search for full-text publications and reports describing economic evaluations of tisagenlecleucel for r/r ALL was conducted in OVID MEDLINE/Embase (March 16, 2023) and HTA databases where English reports were available (March 28, 2023) (a list of HTAs searched is in the Supplemental Methods). Records were excluded if not in English or did not report cost-effectiveness results of tisagenlecleucel in r/r ALL.

PRISMA flow diagram of studies and reports included in the systematic literature review (SLR). HTA health technology assessment, PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses, SLR systematic literature review. a The searches in OVID MEDLINE/Embase and the HTA databases were conducted on March 16, 2023 and March 28, 2023, respectively. b Records were excluded if results were not separately reported for pediatric patients with relapsed/refractory acute lymphoblastic leukemia. c Records were excluded if they were not full-text articles or HTA reports (e.g., note, letter, comment, case report, editorial, protocol, review, meta-analysis, or conference abstract). d Records were excluded if results were not separately reported for the cost-effectiveness analysis. e Records were excluded if the incremental cost-effectiveness ratio of tisagenlecleucel was not reported or was not publicly available

The following data elements were extracted: country, currency, perspectives; patient population; comparators; sources of clinical evidence; model structures; methods of long-term extrapolation of efficacy data; time horizon and discount rates; and outcomes including costs, life years (LYs), quality-adjusted life years (QALYs), and incremental cost-effectiveness ratios (ICERs). The estimated overall survival (OS) rates with tisagenlecleucel in the first 5 years were extracted when available, which were compared with those observed using 5-year ELIANA data.

To assess the results across multiple countries, the ICERs of tisagenlecleucel versus comparators in the base-case and sensitivity/scenario analyses were compared against the respective willingness-to-pay (WTP) thresholds in each country.

2.2 Assessing the Economic Value of Tisagenlecleucel Based on Long-term Data and Improved Infusion Rate

To assess the impact of long-term data, the CEA model reported in Wakase et al. [ 14 ] was updated using 5-year ELIANA data and the infusion rate informed by real-world practice. Wakase et al. was selected because the model was accessible to the authors for update. It was a partitioned survival model (PSM) evaluating the cost effectiveness of tisagenlecleucel in comparison with blinatumomab and clofarabine combination therapy for the treatment of r/r ALL from the perspective of a Japanese public health care payer. A leading decision-tree was applied to the tisagenlecleucel arm, which was separately followed by patients who received tisagenlecleucel infusion versus those who did not receive infusion for various reasons. Among those who received infusion, the efficacy, cost, and disutility inputs were based on the tisagenlecleucel-infused population. Patients who did not receive tisagenlecleucel infusion were assumed to have received blinatumomab, and the associated efficacy and costs were assumed to be the same as those directly assigned to blinatumomab.

All patients were distributed across the following partitioned health states: event-free survival (EFS), progressive disease, and death. The utilities were assumed to depend on health states only and to be independent of treatment. For patients who received tisagenlecleucel infusion, the observed OS and EFS data during the trial period were used to model OS and EFS until Year 5. For comparators, the observed OS during the trial period reported in respective trial publications was used, and then after the end of trial observation period the hazard ratios (HRs) versus tisagenlecleucel derived from the matching-adjusted indirect comparison (MAIC) analyses were used to project OS up to Year 5. EFS data were not publicly available and were thus estimated based on OS data, assuming a constant cumulative HR between OS and EFS. At the end of Year 5, living patients were assumed to be long-term survivors who would experience no additional relapses and whose risk of death was based on standard mortality ratio (SMR)-adjusted general population mortality. The model also assumed that the clinical benefits of subsequent hematopoietic stem cell transplant (HSCT) were captured in the EFS and OS estimations for all treatments as a result of using the direct trial data, whereas the costs and disutilities of the procedure were added separately. The base-case analysis was performed over a lifetime horizon with a monthly model cycle, with costs and effectiveness discounted 2% annually following the recommendations from the Japan HTA agency Center for Outcomes Research and Economic Evaluation for Health (C2H) [ 15 ].

The OS and EFS data originally used by Wakase et al. [ 14 ] for tisagenlecleucel were based on pooled trial data from an earlier data cut with a shorter follow-up time of ELIANA (ClinicalTrials.gov Identifier: NCT02435849), the Phase II ENSIGN trial (NCT02228096), and the Phase I/II B2101J study (NCT01626495). All three trials were single-arm studies without randomization and the enrolled patients had similar characteristics. These three trials enrolled pediatric or young adult patients (aged up to 25 years) with ALL who were primary refractory, chemo-refractory, relapsed after HSCT, chemotherapy resistant, or were otherwise ineligible for HSCT; 44–63% of patients had received prior HSCT and the number of prior regimens ranged from 1 to 9. In the updated model, the observed OS and EFS data were replaced with the 5-year ELIANA data. The longer follow-up data from ELIANA were immature for assessment of the cure assumption, thus the updated model kept the original assumption that patients alive after 5 years were long-term survivors, which was supported by literature [ 16 , 17 , 18 ]. The proportion of patients receiving tisagenlecleucel infusion in the model by Wakase et al. was updated from the trial-based to the real-world infusion rates of 92.5% and 96.0% (using the manufacturer success rate as the upper limit) [ 2 ], reflecting improvements in manufacturing and CAR-T delivery. All other CEA parameters remained the same as those in the original model. LYs, QALYs, and ICERs in the base-case analysis were estimated from the updated model constructed in Excel (Microsoft, Redmond, WA, USA). In addition, deterministic sensitivity analyses (DSAs) and probabilistic sensitivity analyses (PSAs) were performed again to reflect the updated model.

3.1 SLR of CEAs of Tisagenlecleucel for r/r ALL

3.1.1 search results.

A total of 219 publications and 18 HTA reports were identified and screened for eligibility (Fig. 1 ). Sixteen records corresponding to 15 unique studies were included in the SLR, comprising 11 publications [ 14 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 ] and 5 HTA reports [ 29 , 30 , 31 , 32 , 33 ]. Whittington et al. [ 28 ] reflected the same results as a 2018 report from the Institute for Clinical and Economic Review [ 30 ] and were consolidated (henceforth referred to as the ICER report). The other HTA reports included a 2021 report from the C2H in Japan [ 33 ], a 2019 report from the Canadian Agency for Drugs and Technologies in Health (CADTH) [ 29 ], a 2018 report from the National Institute for Health and Care Excellence (NICE) in the UK [ 31 ], and a 2019 report from Scottish Medicines Consortium (SMC) [ 32 ]. The 2019 report from the Medical Services Advisory Committee (MSAC) in Australia [ 34 ] was excluded because ICER results were not publicly available.

3.1.2 Overview of the Included Studies

Three studies were conducted in the USA (Sakar et al. [ 26 ], Lin et al. [ 27 ], and ICER [ 28 , 30 ]); two in the UK (NICE [ 31 ] and SMC [ 32 ]); two in Canada (Furzer et al. [ 25 ] and CADTH [ 29 ]); two in Japan (Wakase et al. [ 14 ] and C2H [ 33 ]); and one each in Australia (Gye et al. [ 19 ]), Ireland (Carey et al. [ 20 ]), Singapore (Wang et al. [ 21 ]), Switzerland (Moradi-Lakeh et al. [ 22 ]), the Netherlands (Thielen et al. [ 23 ]), and Spain (Ribera-Santasusana et al. [ 24 ]) (Table S1 ). Eleven studies considered pediatric and young adult patients up to age 23–25 years, and 4 studies only considered pediatric patients with r/r ALL.

All evaluations were conducted from the health care system perspectives of the respective countries, either nationalized or private. The reports from the Netherlands and USA (Sarkar et al.) also included the societal perspective. As comparators with tisagenlecleucel, 11 studies included blinatumomab, 7 included SC, 7 included clofarabine (as combination therapy [Clo-C], monotherapy [Clo-M], or both), and 1 included inotuzumab ozogamicin. Salvage chemotherapy was typically based on fludarabine, cytarabine, and idarubicin (FLA-IDA). Additionally, the comparator in two evaluations was the regional SOC, defined in Furzer et al. as any treatment a tisagenlecleucel-eligible patient received at their physician’s discretion, including chemotherapy with follow-up HSCT, and in Sarkar et al. as Clo-C followed by HSCT.

The model structure in 8 studies was a PSM preceded by a decision tree. The PSM part reflects the health states of EFS or PFS, progressed/relapsed disease, and death. Lin et al. used a Markov structure and the ICER report used a semi-Markov structure preceded by a decision tree. Thielen et al., Ribera-Santasusana et al., and the CADTH report used a PSM without a decision tree, while Furzer et al. and Sarkar et al. used a microsimulation approach.

3.1.3 Efficacy Sources for Tisagenlecleucel and Long-term OS Extrapolation Approaches

The majority of studies (12) used data from ELIANA, ENSIGN, and the B2101J study (Table 1 ). Two evaluations used data from ELIANA and ENSIGN (Gye et al. and Carey et al.), and one (Sarkar et al.) used ELIANA data only. In addition, in the CADTH report, the company submission used pooled data from all 3 sources, but the re-analysis by CADTH used only ELIANA data. Nine evaluations used efficacy inputs per the tisagenlecleucel-infused populations in clinical trials, 3 used efficacy inputs from the intent-to-treat (ITT) population, and 3 did not report this information.

Among the 15 studies, before applying the long-term survival extrapolation, 7 explored standard parametric models, spline models, and mixture cures, whereas 3 only explored standard parametric models and 5 only used observed data. Typically, statistical fit, visual fit, and/or clinical assessment were used to select the best fitting model. All studies used a lifetime horizon (i.e., 70–88 years). In 13 studies, SMRs were applied to extrapolate long-term survival after 3–5 years. The majority of studies used a SMR of 9.1 sourced from a Canadian cohort study in childhood cancer patients who had survived at least 5 years [ 17 ], whereas 3 studies used SMRs of 15.5, 15.2 and 20.8, respectively, obtained from studies investigating ALL survivors specifically [ 17 , 35 , 36 ].

3.1.4 Overall Survival Estimates for Tisagenlecleucel in the Identified CEAs Compared with 5-year ELIANA Data

The OS rates for tisagenlecleucel in the first 5 years, as predicted by the included evaluations and as observed in ELIANA with 5-year data, are summarized in Table 2 . Overall, the 3- and 5-year OS rates were available and extracted from 8 of the 15 studies. Of these 8 studies, 3 reported the OS rates among the ITT population and 5 reported the OS rates among infused patients only.

In the 5-year ELIANA data, over 30% of patients were still under follow-up at Year 5. The number of patients at risk of death among the ITT population ( n = 98) was 66, 43, and 33 at Years 1, 3, and 5, respectively; among infused patients ( n = 80), the numbers were 61, 43, and 25, respectively. The survival rate of infused patients in the 5-year ELIANA data was higher than that of infused patients in the previous data cut used in the CEA by Wakase et al. (3-year OS: 63.8% vs 62.8%; EFS: 49.2% vs 47.6%).

Across all 8 studies with available information, the 3- and 5-year OS rates among infused patients (ranging from 18.8 to 63.4% at Year 3 and from 7.2 to 44.2% at Year 5) were lower than those observed in the ELIANA 5-year data, which were 63.8% and 56.0%, respectively. After applying the real-world infusion rate of 92.5–96.0% to the infused efficacy observed in the ELIANA 5-year data, the OS rates estimated among all intended for tisagenlecleucel infusion were 59.0–61.3% at Year 3 and 51.8–53.8% at Year 5, higher than those predicted by the included CEAs.

3.1.5 Results of the Economic Evaluations

The results of the included studies’ CEA analyses are summarized in Table 3 . Among the included CEA studies, the efficacy inputs of comparators were generally sourced from the respective clinical trials and the cost components accounted for comparators were similar, including drug acquisition and administration costs, hospitalization costs during treatment, AE costs, disease management costs, and subsequent treatment costs. However, the specific considerations varied across studies. The incremental QALYs for tisagenlecleucel in the base-case analyses ranged widely from − 0.61 in Lin et al. to 10.77 in Thielen et al. In Lin et al., the assumed 5-year relapse-free survival (RFS) was overly conservative at 0%; in Thielen et al., all patients were assumed to receive tisagenlecleucel infusion. The company submission to C2H included incremental QALYs of 8.56–9.55 but was revised in the re-analyses by C2H to 6.64–8.57. The company submission to CADTH included incremental QALYs of 8.61–11.74 but was revised in the re-analyses by CADTH to 7.94–10.60.

Among the US studies, the base-case ICERs per QALY gained ranged from USD $45,871 in the ICER report to $61,000–$184,000 in Lin et al. (Table 3 ). The company submission to C2H estimated generally lower ICERs per QALY gained (¥1,994,592–¥2,087,581) than the re-analysis by C2H (¥2,184,285–¥2,747,550) [ 33 ]. This was potentially due to more conservative assumptions for long-term extrapolation of survival in the C2H re-analysis. The CADTH report estimated generally lower ICERs per QALY gained (CAD $51,295–$54,393) than Furzer et al. (CAD $71,000–$281,000) due to more conservative assumptions for the QALYs for tisagenlecleucel in the latter. The two UK CEAs arrived at similar ICERs per QALY gained, ranging from £18,392–£25,404 in the NICE report and £25,238 in the SMC report.

Overall, 13 of the 15 studies found tisagenlecleucel to be cost effective for r/r ALL under their country-specific WTP thresholds in the base case (Fig. 2 ). Among these 13 studies, Lin et al. and Furzer et al. each included three scenarios assuming different efficacy for tisagenlecleucel: 0%, 20%, and 40% RFS rates in Lin et al. and low cure rates of 10%, 20%, and 40% in Furzer et al. Tisagenlecleucel was projected to be cost effective under the most reasonable, although still conservative, scenarios per its true efficacy (i.e., a 40% RFS rate in Lin et al. and 40% cure rate in Furzer et al.). The two studies that did not find tisagenlecleucel cost effective, Gye et al. and Carey et al., both reported fewer discounted QALYs for tisagenlecleucel than other studies (i.e., 5.36 and 4.33, respectively, vs 8.29–16.76 in other studies when reasonable scenarios were used in Lin et al. and Furzer et al.). This may be due to the higher benefit discount rates used in both studies (5.0% in Gye et al. and 4.0% in Carey et al. vs 1.5–3.5% in others) and a higher SMR in Carey et al. (15.5 vs 9.1 applied in most studies). In addition, Gye et al. and Carey et al. used WTP thresholds of AUD $50,000/USD $33,454 and €45,000/USD $49,352, respectively, which were lower than thresholds for other countries ranging from USD $52,725–$150,000, except Spain (USD $32,901). The WTP thresholds that should be used are subject to debate and historical influence. If twice the per-capita country-specific gross domestic product were instead used as the threshold, all studies found tisagenlecleucel to be cost effective (eFig. 1 ).

Cost effectiveness of tisagenlecleucel for r/r ALL compared with other therapies, based on WTP threshold of the country. The dots represent the comparison of the base-case incremental cost-effectiveness ratio against the respective WTP threshold while the ranges represent the comparison of incremental cost-effectiveness ratios from sensitivity/scenario analyses against the WTP threshold. The red dashed line represents the WTP threshold where incremental cost-effectiveness ratios on the left of this line reflect that tisagenlecleucel was found to be cost-effective at the study country’s WTP for a comparator. C2H Center for Outcomes Research and Economic Evaluation for Health, CADTH Canadian Agency for Drugs and Technologies in Health, Com comparator, ICER Institute for Clinical and Economic Review, NICE National Institute of Health and Care Excellence, RFS relapse-free survival, r/r ALL relapsed/refractory acute lymphoblastic leukemia, SMC Scottish Medicines Consortium, WTP willingness to pay. a In Furzer et al. 2020, a cure state was included to account for the limited long-term survival information currently available for tisagenlecleucel. The base-case estimates used a range of cure rates from 10 to 40% for those offered treatment based on expert opinion. b In Lin et al. 2018, three scenarios that cover a broad range of plausible long-term outcomes on the basis of observed variance and expert opinion were evaluated: 0%, 20% and 40% 5-year RFS rates without hematopoietic stem-cell transplantation and tisagenlecleucel as a bridge to transplantation under a 0% transplantation-free 5-year RFS scenario. c In Lin et al. 2018, tisagenlecleucel was dominated by blinatumomab in the base-case analysis and all scenario analyses for the scenario applying 0% as the 5-year RFS rate. d In Moradi-Lakeh et al. 2021 and Wakase et al. 2022 the lowest sensitivity/scenario analysis result was tisagenlecleucel being dominant over comparators. e In CADTH 2019, only the results of price-reduction scenario analyses were reported. f In SMC 2019, only the results of selected sensitivity analyses were reported. The lowest sensitivity/scenario analysis result was not available

3.2 Economic Value of Tisagenlecleucel Based on Long-term Data and Improved Infusion Rate

We updated the model used in Wakase et al. with the 5-year OS and EFS data observed among tisagenlecleucel-infused patients in ELIANA, which yielded improvements in the modeled OS versus that derived from the pooled tisagenlecleucel data of the prior data cut (Fig. 3 ). When applying the most recent OS projections and an infusion rate of 92.5–96% to the CEA of Wakase et al., the associated QALYs for tisagenlecleucel increased by 3.0–3.4 years and LYs increased by 3.7–4.2 years. The ICERs of tisagenlecleucel decreased from ¥2,035,071 to ¥1,787,988−¥1,789,048 versus blinatumomab and from ¥2,644,702 to ¥2,257,837−¥2,275,181 versus clofarabine combination therapy (table inset, Fig. 3 ). Even if the infusion rate remained unchanged (i.e., 84% based on pooled ELIANA, ENSIGN, and B2101J data), the QALYs and LYs for tisagenlecleucel still increased by 1.9 and 2.5, respectively.

The impact of 5-year ELIANA data and improved infusion rate on the predicted LYs, QALYs and ICERs. ICER incremental cost-effectiveness ratio, LY life year, OS overall survival, QALY quality-adjusted life year

The DSA results of the updated model, reflecting updated efficacy and current infusion rate (eFig. 2 [infusion rate of 96%] and eFig. 3 [infusion rate of 92.5%]), suggested that the ICERs of tisagenlecleucel versus both comparators were reduced compared to the original Wakase et al. model. Similar to the original model, when productivity gain was considered, tisagenlecleucel dominated both comparators. The largest ICER across all DSAs decreased from ¥2,885,485 to ¥2,549,724- ¥2,551,493 versus blinatumomab and decreased from ¥3,756,251 to ¥3,223,428- ¥3,248,597 versus clofarabine combination therapy. Across all other DSAs, the ICER were reduced and the degree of reduction ranged from ¥142,113 to ¥594,428 versus blinatumomab and from ¥242,491 to ¥971,266 versus clofarabine combination therapy. At a threshold of ¥7,500,000 per QALY gained [ 37 , 38 ], the probability of tisagenlecleucel being cost effective remained 100% compared to both comparators in the PSA analysis of the updated model, considering either rate of infusion (eFig. 4 [infusion rate of 96%] and eFig. 5 [infusion rate of 92.5%]).

4 Discussion

In the SLR of prior economic analyses of tisagenlecleucel for r/r ALL, there were large variations in the results that were partially attributable to the differences in the extrapolation approach used to project long-term efficacy. All studies used some type of extrapolation mechanism based on trial data, particularly ELIANA, but some incorporated a cure assumption for patients in remission at 5 years while others did not. Additionally, some assumptions explored were overly conservative relative to the observed long-term data, such as low cure rates or RFS. This heterogeneity is reflected in the wide ranges of LYs (7.13–20.60) and QALYs (2.96–16.76) of tisagenlecleucel reported by the studies, which can greatly impact the value assessment. The variations in the models’ efficacy inputs largely stemmed from the lack of long-term efficacy data for tisagenlecleucel and differing willingness to accept uncertainties for new CAR-T therapies.

Although the majority of CEAs found that tisagenlecleucel was cost effective under their country’s WTP, its value was likely still underestimated in many studies. Specifically, both the LYs and QALYs for tisagenlecleucel substantially increased when applying efficacy data from the 5-year ELIANA data cut to the model of Wakase et al., whether or not improvements in the CAR-T infusion rates were considered. Notably, the 5-year ELIANA data revealed that the best extrapolation approach to predict the observed OS of tisagenlecleucel was to fit all standard parametric survival models and spline models and then calculate the average weighted by Akaike information criterion of each model. This approach incorporates spline models, which can provide more flexibility in curve fitting compared to standard parametric survival models and may better reflect long-term extrapolation.

Real-world evidence has consistently confirmed the efficacy of tisagenlecleucel in r/r ALL as observed in ELIANA. For example, a 2022 real-world study reported that the 1-year OS of 185 patients with r/r B-ALL infused with commercial tisagenlecleucel was 72% [ 39 ]. The Center for International Blood and Marrow Transplant Research (CIBMTR) registry of patients with r/r B-ALL reported a 3-year OS rate of 58.7% among 578 patients infused with commercial tisagenlecleucel, but the safety outcomes were more favorable than in ELIANA [ 40 ]. This finding is notable given that the real-world CIBMTR population is broader and more diverse than that of ELIANA, which excluded patients under the age of 3 years or with < 5% bone marrow blasts.

In a recent (2023) public summary by MSAC for the review of tisagenlecleucel in r/r ALL [ 41 ], released after the conduct of the present SLR, MSAC considered that the economic model had overestimated the benefits of tisagenlecleucel and underestimated the value of blinatumomab. The statement was partly due to the fact that data used for tisagenlecleucel and blinatumomab in the model submitted to MSAC were directly derived from the respective clinical trials without accounting for cross-trial differences in patient populations. Conversely, MAIC of tisagenlecleucel versus comparators was used in the Wakase CEA and our updated analysis. In addition, the statement was in the context of the comparison between tisagenlecleucel and blinatumomab. However, based on the OS rate in infused patients, the 30-month OS rate per 5-year ELIANA data (65.3%; November 2022 data cutoff) was higher than that reported in the MSAC document based on prior data cuts of ELIANA and ENSIGN data (55.11% and 59.64%, respectively, from ELIANA December 2017 and ENSIGN September 2022 data cutoffs); this is consistent with our findings in the present study that the value of tisagenlecleucel was likely underestimated when using the prior data cut. MSAC also considered that some of the benefit of tisagenlecleucel could be influenced by the use of subsequent treatments, particularly HSCT. However, subsequent HSCT was considered as part of the treatment strategy in clinical practice for some patients receiving tisagenlecleucel and was allowed in ELIANA. Therefore, assessments for tisagenlecleucel should reflect both the benefit as well as costs and disutilities caused by subsequent HSCT, which was the case in the Wakase et al. CEA, our updated analysis, and the MSAC submission. The real-world evidence from the CIBMTR registry suggests that tisagenlecleucel is also effective without subsequent HSCT [ 42 ]. The use of subsequent HSCT after tisagenlecleucel in current clinical practice is driven partially by efficacy and partially by existing practice habit and belief, which may change in the future.

These results demonstrate the importance of long-term data and the challenge of projecting outcomes with limited observed data for novel therapies like CAR-T, where both manufacturing and real-world clinical practices will evolve. Further, our findings underscore the need to evaluate the appropriate timing to derive value-based pricing for novel therapies, as the results from CEAs based on early trial data could be uncertain.

Nevertheless, coverage and reimbursement decisions for one-time, potentially curative therapy require balancing the needs of diverse stakeholders. These include patients’ urgent need for a potentially life-saving therapy, payers’ needs for robust evidence to quantify the treatment’s value, and the manufacturer’s need to preserve the value and sustainable innovation of novel medical technologies. Irrespective of the selected reimbursement approaches, it is important for policy makers to promptly establish interim reimbursement policies to ensure that patients have timely access to new treatments while allowing long-term data and real-world evidence on both efficacy and safety to be continuously collected. Periodic re-assessment could help ensure that the scarce resources are used wisely to cover treatments that provide value to society. As new data emerge, reimbursement decisions can be revised accordingly to better reflect the demonstrated value of treatments. Additionally, reassessments could encourage more detailed safety and efficacy monitoring post-approval, which could be of great benefit to patients and to the development of future therapies. By experimenting with flexible approaches such as outcome-based contracts, we can mitigate uncertainties when long-term data are not available while shortening the gap between the time of regulatory approval and patients’ receipt of treatments.

The results of this study are subject to several limitations. First, the impact of long-term efficacy data and the real-world infusion rate of tisagenlecleucel was only assessed using the model by Wakase et al., as that was the only model available to the authors. Similar or potentially larger impacts could be expected on other models, as the OS estimates in Wakase et al. were closer to the values observed in the long-term ELIANA data whereas other studies tended to be more conservative. Second, the current studies found that prior CEAs made conservative assumptions for tisagenlecleucel, although this does not imply that conservative assumptions were used in CEAs of other novel therapies. Third, this study utilized conventional cost-effectiveness thresholds when a case can be made for higher thresholds for rare, health-catastrophic conditions such as r/r ALL [ 43 ].

5 Conclusions

Using tisagenlecleucel as a case study, we discovered that most prior CEAs included conservative assumptions regarding its long-term efficacy yet found it cost effective for r/r ALL. The application of long-term clinical and real-world data of tisagenlecleucel’s efficacy and improved infusion rate increased the projected LYs and QALYs gained, further supporting that it is a cost-effective option for r/r ALL. Considering the unique nature of one-time, potentially curative therapies like CAR-T and the insights gained from this case study, it is important for payers to promptly establish flexible, value-based reimbursement policies to balance the need for timely access to novel treatments with uncertainty in initial valuations.

Frieden TR. Evidence for health decision making - beyond randomized, controlled trials. N Engl J Med. 2017;377(5):465–75.

Article   PubMed   Google Scholar  

Rodrigues M, Duran E, Eschgfaeller B, Kuzan D, Habucky K. Optimizing commercial manufacturing of tisagenlecleucel for patients in the US: a 4-year experiential journey. Blood. 2021;138(Supplement 1):1768.

Article   Google Scholar  

Rodrigues M, Kuzan D, Yallouridis A, Saffar J-M, Eschgfaeller B. Commercial manufacturing experience of tisagenlecleucel in Europe: > 3 years journey. 48th annual meeting of the European Society for Blood and Marrow Transplantation (EBMT); March 19-23, 2022; Prague, Czech Republic.

Mullard A. FDA approves first CAR T therapy. Nat Rev Drug Discov. 2017;16(10):669.

PubMed   Google Scholar  

United States Food and Drug Administration. Highlights of prescribing information: KYMRIAH (tisagenlecleucel) 2017. https://www.fda.gov/media/107296/download . Accessed 1 June 2023.

Jeha S, Gaynon PS, Razzouk BI, Franklin J, Kadota R, Shen V, et al. Phase II study of clofarabine in pediatric patients with refractory or relapsed acute lymphoblastic leukemia. J Clin Oncol. 2006;24(12):1917–23.

Article   CAS   PubMed   Google Scholar  

Siegel RL, Miller KD, Fuchs HE, Jemal A. Cancer statistics, 2021. CA Cancer J Clin. 2021;71(1):7–33.

Pui CH, Carroll WL, Meshinchi S, Arceci RJ. Biology, risk stratification, and therapy of pediatric acute leukemias: an update. J Clin Oncol. 2011;29(5):551–65.

Maude SL, Laetsch TW, Buechner J, Rives S, Boyer M, Bittencourt H, et al. Tisagenlecleucel in children and young adults with B-cell lymphoblastic leukemia. N Engl J Med. 2018;378(5):439–48.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Miliotou AN, Papadopoulou LC. CAR T-cell therapy: a new era in cancer immunotherapy. Curr Pharm Biotechnol. 2018;19(1):5–18.

Shimabukuro-Vornhagen A, Böll B, Schellongowski P, Valade S, Metaxa V, Azoulay E, et al. Critical care management of chimeric antigen receptor T-cell therapy recipients. CA Cancer J Clin. 2022;72(1):78–93.

Shi X, Wu H. Recent advances in the prevention and management of cytokine release syndrome after chimeric antigen receptor T-cell therapy. Eur J Inflamm. 2022. https://doi.org/10.1177/1721727X221078727 .

Ghassemi S, Durgin JS, Nunez-Cruz S, Patel J, Leferovich J, Pinzone M, et al. Rapid manufacturing of non-activated potent CAR T cells. Nat Biomed Engin. 2022;6(2):118–28.

Article   CAS   Google Scholar  

Wakase S, Teshima T, Zhang J, Ma Q, Watanabe Y, Yang H, et al. Cost-effectiveness analysis of tisagenlecleucel for the treatment of pediatric and young adult patients with relapsed or refractory B cell acute lymphoblastic leukemia in Japan. Transplant Cell Ther. 2021;27(3):241 ( e1-e11 ).

Japan Center for Outcomes Research and Economic Evaluation for Health- National Institute of Public Health (C2H). Guideline for preparing cost-effectiveness evaluation to the Central Social Insurance Medical Council, v3.0. January 19, 2022. https://c2h.niph.go.jp/tools/guideline/guideline_en.pdf . Accessed 8 Nov 2023.

Centre for Health Technology Evaluation- National Institute for Health and Care Excellence (NICE). Exploring the assessment and appraisal of regenerative medicines and cell therapy products 2016. https://www.nice.org.uk/media/default/about/what-we-do/science%20policy%20and%20research/regenerative-medicine-study-march-2016.pdf . Accessed 19 Feb 2024.

MacArthur AC, Spinelli JJ, Rogers PC, Goddard KJ, Abanto ZU, McBride ML. Mortality among 5-year survivors of cancer diagnosed during childhood or adolescence in British Columbia, Canada. Pediatr Blood Cancer. 2007;48(4):460–7.

Mody R, Li S, Dover DC, Sallan S, Leisenring W, Oeffinger KC, et al. Twenty-five-year follow-up among survivors of childhood acute lymphoblastic leukemia: a report from the Childhood Cancer Survivor Study. Blood. 2008;111(12):5515–23.

Gye A, Goodall S, De Abreu LR. Cost-effectiveness analysis of tisagenlecleucel versus blinatumomab in children and young adults with acute lymphoblastic leukemia: partitioned survival model to assess the impact of an outcome-based payment arrangement. Pharmacoeconomics. 2023;41(2):175–86.

Carey N, Leahy J, Trela-Larsen L, McCullagh L, Barry M. Tisagenlecleucel for relapsed/refractory acute lymphoblastic leukemia in the Irish healthcare setting: cost-effectiveness and value of information analysis. Int J Technol Assess Health Care. 2022;38(1): e56.

Wang XJ, Wang YH, Ong MJC, Gkitzia C, Soh SY, Hwang WYK. Cost-effectiveness and budget impact analyses of tisagenlecleucel in pediatric and young adult patients with relapsed or refractory B-cell acute lymphoblastic leukemia from the Singapore healthcare system perspective. Clinicoecon Outcomes Res. 2022;14:333–55.

Moradi-Lakeh M, Yaghoubi M, Seitz P, Javanbakht M, Brock E. Cost-effectiveness of tisagenlecleucel in paediatric acute lymphoblastic leukaemia (pALL) and adult diffuse large B-Cell lymphoma (DLBCL) in Switzerland. Adv Ther. 2021;38(6):3427–43.

Thielen FW, van Dongen-Leunis A, Arons AMM, Ladestein JR, Hoogerbrugge PM, Uyl-de Groot CA. Cost-effectiveness of anti-CD19 chimeric antigen receptor T-cell therapy in pediatric relapsed/refractory B-cell acute lymphoblastic leukemia. A societal view. Eur J Haematol. 2020;105(2):203–15.

Ribera Santasusana JM, de Andres Saldana A, Garcia-Munoz N, Gostkorzewicz J, Martinez Llinas D, Diaz de Heredia C. Cost-effectiveness analysis of tisagenlecleucel in the treatment of relapsed or refractory B-cell acute lymphoblastic leukaemia in children and young adults in Spain. Clinicoecon Outcomes Res. 2020;12:253–64.

Article   PubMed   PubMed Central   Google Scholar  

Furzer J, Gupta S, Nathan PC, Schechter T, Pole JD, Krueger J, et al. Cost-effectiveness of tisagenlecleucel vs standard care in high-risk relapsed pediatric acute lymphoblastic leukemia in Canada. JAMA Oncol. 2020;6(3):393–401.

Sarkar RR, Gloude NJ, Schiff D, Murphy JD. Cost-effectiveness of chimeric antigen receptor T-cell therapy in pediatric relapsed/refractory B-cell acute lymphoblastic leukemia. J Natl Cancer Inst. 2019;111(7):719–26.

Lin JK, Lerman BJ, Barnes JI, Boursiquot BC, Tan YJ, Robinson AQL, et al. Cost effectiveness of chimeric antigen receptor T-cell therapy in relapsed or refractory pediatric B-cell acute lymphoblastic leukemia. J Clin Oncol. 2018;36(32):3192–202.

Whittington MD, McQueen RB, Ollendorf DA, Kumar VM, Chapman RH, Tice JA, et al. Long-term survival and value of chimeric antigen receptor T-cell therapy for pediatric patients with relapsed or refractory leukemia. JAMA Pediatr. 2018;172(12):1161–8.

Canadian Agency for Drugs and Technologies in Health (CADTH). Tisagenlecleucel for acute lymphoblastic leukemia and diffuse large B-cell lymphoma: Economic review report. CADTH Optimal Use Report, vol. 8. no. 3e. 2019. https://www.cadth.ca/sites/default/files/pdf/car-t/op0538-tisagenlecleucel-economic-report-DLBCL-jan2019.pdf . Accessed 21 May 2023.

Institute for Clinical and Economic Review (ICER). Chimeric antigen receptor T-cell therapy for B- cell cancers: Effectiveness and value. Final evidence report. 2018. https://icer.org/wp-content/uploads/2020/10/ICER_CAR_T_Final_Evidence_Report_032318.pdf . Accessed 21 May 2023.

National Institute for Health and Care Excellence (NICE). Tisagenlecleucel for treating relapsed or refractory B-cell acute lymphoblastic leukaemia in people aged up to 25 years. Technology appraisal guidance. 2018. https://www.nice.org.uk/guidance/ta554/resources/tisagenlecleucel-for-treating-relapsed-or-refractory-bcell-acute-lymphoblastic-leukaemia-in-people-aged-up-to-25-years-pdf-82607021872837 . Accessed 21 May 2023.

Scottish Medicines Consortium (SMC). Tisagenlecleucel 1.2 x 106 – 6 x 108 cells dispersion for infusion (Kymriah ® ). Novartis Pharmaceuticals UK Ltd. 2019. https://www.scottishmedicines.org.uk/media/4228/tisagenlecleucel-kymriah-final-feb-2019-amended-7319-for-website.pdf . Accessed 21 May 2023.

Center for Outcomes Research and Economic Evaluation for Health. C2H Evaluation Report: C2H1902. Tisagenlecleucel /B-ALL (Kymriah). https://c2h.niph.go.jp/results/C2H1902/C2H1902_Report_BALL_Eng.pdf . Accessed 19 Feb 2024.

Medical Services Advisory Committee. 1519 - Tisagenlecleucel (CTL019) for treatment of refractory CD19-positive leukaemia and lymphoma. http://www.msac.gov.au/internet/msac/publishing.nsf/Content/1519-public . Accessed 19 Feb 2024.

Fidler MM, Reulen RC, Winter DL, Kelly J, Jenkinson HC, Skinner R, et al. Long term cause specific mortality among 34 489 five year survivors of childhood cancer in Great Britain: population based cohort study. BMJ. 2016;354: i4351.

Armstrong GT, Chen Y, Yasui Y, Leisenring W, Gibson TM, Mertens AC, et al. Reduction in late mortality among 5-year survivors of childhood cancer. N Engl J Med. 2016;374(9):833–42.

Hasegawa M, Komoto S, Shiroiwa T, Fukuda T. Formal implementation of cost-effectiveness evaluations in Japan: a unique health technology assessment system. Value Health. 2020;23(1):43–51.

Medical Economics Division-Health Insurance Bureau, Ministry of Health Labour and Welfare. Full scale introduction of cost-effectiveness evaluations in Japan. Overview (2/20/2019). https://c2h.niph.go.jp/tools/system/overview_en.pdf . Accessed 15 May 2020.

Schultz LM, Baggott C, Prabhu S, Pacenta HL, Phillips CL, Rossoff J, et al. Disease burden affects outcomes in pediatric and young adult B-cell lymphoblastic leukemia after commercial tisagenlecleucel: a pediatric real-world chimeric antigen receptor consortium report. J Clin Oncol. 2022;40(9):945–55.

John S, Heim M, Curran KJ, Hall EM, Keating AK, Baumeister SH, et al. Improved relapse-free survival (RFS) for pedriatric and young adult patients with relapsed or refractory (r/r) B-cell acute lymphoblastic leukemia (B-ALL) and low or intermediate preinfusion disease burden treated with tisagenlecleucel: Results from the CIBMTR registry. Tandem Transplantation & Cellular Therapy Meeting of ASTCT and CIBMTR; February 15-19, 2023; Orlando, FL, US.

Medical Services Advisory Committee. Application No. 1748—Review of tisagenlecleucel (Kymriah ® ) for treatment of acute lymphoblastic leukaemia in paediatric and young adult patients (pALL) 2023. http://www.msac.gov.au/internet/msac/publishing.nsf/Content/D900A0D953425184CA25895C0017A252/$File/1748%20Final%20PSD%20-%20July%202023%20(redacted).pdf . Accessed 19 Feb 2024.

John S, Heim M, Curran K, Hall E, Keating A, Baumeister S. Improved relapse-free survival (RFS) for pedriatric and young adult patients with relapsed or refractory (r/r) B-cell acute lymphoblastic leukemia (B-ALL) and low or intermediate preinfusion disease burden treated with tisagenlecleucel: Results from the CIBMTR registry. Tandem Transplantation & Cellular Therapy Meeting of ASTCT and CIBMTR; February 15-19, 2023; Orlando, FL.

Garrison LP, Jackson T, Paul D, Kenston M. Value-based pricing for emerging gene therapies: the economic case for a higher cost-effectiveness threshold. J Manag Care Spec Pharm. 2019;25(7):793–9.

Download references

Acknowledgements

Medical writing assistance was provided by Shelley Batts, Ph.D., an independent contractor of Analysis Group, Inc., and funded by Novartis.

Author information

Authors and affiliations.

Children’s Hospital of Philadelphia, Philadelphia, PA, USA

Theodore Laetsch

Novartis Pharmaceuticals Corporation, East Hanover, NJ, USA

Analysis Group, Inc., Boston, MA, USA

Hongbo Yang

Analysis Group, Inc., Beijing, China

Yanwen Xie & Dudan Zhang

School of Pharmacy, University of Washington, Seattle, WA, USA

Louis Garrison

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Hongbo Yang .

Ethics declarations

This study was supported by Novartis.

Conflict of interest

Theodore Laetsch holds stock in Advanced Microbubbles; has consulting or advisory roles with Novartis, Aptitude Health, Jumo Health, Massive Bio, Medscape, AI Therapeutics, Jazz Pharmaceuticals, GentBio, Menarini, Pyramid Biosciences, Targeted Oncology, and Treeline Biosciences; and research funding from Lilly, Roche/Genentech, Taiho Oncology, Advanced Accelerator Applications/Novartis, Bristol-Myers Squibb, BioAtla, Hoffman-LaRoche, Pfizer, Bayer, and Turning Point Therapeutics. Louis Garrison reports having received consulting fees in the last 2 years for other research activities with a number of biopharmaceutical companies, including Novartis Gene Therapy, Pfizer, Astra-Zeneca, GSK, MSD, Eli Lilly, Genentech, Roche Molecular Systems, BioMarin, and UniQure. Jie Zhang is an employee of Novartis and owns stock/options. Hongbo Yang, Yanwen Xie, and Dudan Zhang are employees of Analysis Group, Inc., which has received consulting fees from Novartis.

Availability of data and material

The data supporting this study were derived from publicly available literature (CEAs) or are contained in the article. Individual patient data from ELIANA will not be shared due to patient privacy.

Ethics approval

This was systematic literature review and economic analysis using public or previously collected, anonymized data. Thus, no ethical review was required. The study was conducted in accordance with the 1964 Declaration of Helsinki and its amendments.

Consent to participate

Consent to publish, code availability.

The model described in this study will not be publicly shared due to confidential information.

Author contributions

All authors designed the study. HY, YX, and DZ performed the literature review and CEA analysis. All authors participated in revising the manuscript and have read and approved the final version.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 1962 kb)

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/ .

Reprints and permissions

About this article

Laetsch, T., Zhang, J., Yang, H. et al. Evolving Evidence-Based Value Assessment of One-Time Therapies: Tisagenlecleucel as a Case Study. Appl Health Econ Health Policy (2024). https://doi.org/10.1007/s40258-024-00882-4

Download citation

Accepted : 26 March 2024

Published : 29 April 2024

DOI : https://doi.org/10.1007/s40258-024-00882-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Find a journal
• Publish with us
• Track your research

Asking the better questions that unlock new answers to the working world's most complex issues.

Trending topics

AI insights

EY Center for Board Matters

EY podcasts

EY webcasts

Operations leaders

Technology leaders

EY helps clients create long-term value for all stakeholders. Enabled by data and technology, our services and solutions provide trust through assurance and help clients transform, grow and operate.

EY.ai - A unifying platform

Strategy, transaction and transformation consulting

Technology transformation

Tax function operations

Climate change and sustainability services

EY Ecosystems

EY Nexus: business transformation platform

Discover how EY insights and services are helping to reframe the future of your industry.

Case studies

How Mojo Fertility is helping more men conceive

26-Sep-2023 Lisa Lindström

Strategy and Transactions

How a cosmetics giant’s transformation strategy is unlocking value

13-Sep-2023 Nobuko Kobayashi

How a global biopharma became a leader in ethical AI

15-Aug-2023 Catriona Campbell

We bring together extraordinary people, like you, to build a better working world.

Experienced professionals

EY-Parthenon careers

Student and entry level programs

Talent community

At EY, our purpose is building a better working world. The insights and services we provide help to create long-term value for clients, people and society, and to build trust in the capital markets.

Press release

Extreme E and EY publish Season 3 report, recording 8.2% carbon footprint reduction as female-male performance gap continues to narrow

09-Apr-2024 Michael Curtis

EY announces acceleration of client AI Business Model adoption with NVIDIA AI

20-Mar-2024 Barbara Dimajo

EY announces launch of artificial intelligence platform EY.ai following US$1.4b investment

13-Sep-2023 Rachel Lloyd

No results have been found

Recent Searches

How do you steady the course of your IPO journey in a changing landscape?

EY Global IPO Trends Q1 2024 provides insights, facts and figures on the IPO market and implications for companies planning to go public. Learn more.

How can the moments that threaten your transformation define its success?

Leaders that put humans at the center to navigate turning points are 12 times more likely to significantly improve transformation performance. Learn More.

Artificial Intelligence

EY.ai - a unifying platform

Select your location

close expand_more

EY Assess: maturity assessment platform for digital, cybersecurity and risk

Rapidly assess maturity domains such as innovation, digital, cybersecurity and risk with EY Assess.

EY Assess is an all-in-one maturity assessment tool. Its self-assessment capabilities, real-time reporting and AI-based evidence analysis help enable organizations to make informed decisions, benchmark against industry standards, and improve maturity.

Accelerating your transformation journey

EY Assess is a key component of the EY Transformation Platform, a central platform designed to streamline transformation journeys, forging a digital pathway from strategy to execution. The EY Transformation Platform supports organizations to unlock their full potential, establishing clear objectives and harnessing data for actionable insights.

Foster consistent assessments

Foster organizational maturity by evaluating critical areas, including identifying and addressing weaknesses.

Optimize fact-based decisions

Facilitates quick decision-making by providing real-time insights into the organization's maturity progress.

Accelerate time-to-insight

Advances outputs and builds additional insight by leveraging the power of state-of-the-art AI capabilities.

Maturity assessments

• Create broadly configurable questionnaires in the form of maturity assessments or surveys.
• Assess and diagnose maturity across various critical areas, including transformation health, digital, innovation, cyber and risk.
• Streamline evidence capture and assess progress with self-assessment capabilities to facilitate efficient tracking and documentation.

Dynamic reporting and dashboard

• Discover new meaningful insights and relevant scores to make data-driven decisions.
• Enhance reporting to analyze results by leveraging multiple methodologies.
• Compare your questionnaires’ results with previous versions (e.g., the last one, or from three or five years ago) to better review and track progress over time.

Benchmarking

• Assess and benchmark your organization's performance against relevant standards with industry and peer benchmarking tools, providing valuable insights.
• Uncover performance gaps in people, process, data and technology-related areas with AI-based evidence analysis, enhancing assessment insights.
• Customize benchmarks and compare and analyze results instantly using an interactive dashboard for actionable insights and informed decision-making.

Case study: Cyber capabilities assessment

The cyber capabilities of three subsidiaries of a global TMT company were evaluated by the EY organization, selected for our extensive experience in cybersecurity and assessment.

Case study: Annual assessment and benchmarking

Defined the strategic direction for cloud security to oversee secure deployment of technology in the cloud for a global building material retailer.

Request a demo today

Start your transformation journey with EY Assess and other products powered by the EY Transformation Platform.

Related Transformation Products

• EY Transformation Platform
• EY CogniStreamer
• EY Trend Sighter

• Connect with us
• Our locations
• Legal and privacy
• Open Facebook profile
• Open X profile
• Open LinkedIn profile
• Open Youtube profile

EY refers to the global organization, and may refer to one or more, of the member firms of Ernst & Young Global Limited, each of which is a separate legal entity. Ernst & Young Global Limited, a UK company limited by guarantee, does not provide services to clients.

• Open access
• Published: 22 April 2024

Artificial intelligence and medical education: application in classroom instruction and student assessment using a pharmacology & therapeutics case study

• Kannan Sridharan 1 &
• Reginald P. Sequeira 1  

BMC Medical Education volume  24 , Article number:  431 ( 2024 ) Cite this article

544 Accesses

2 Altmetric

Metrics details

Artificial intelligence (AI) tools are designed to create or generate content from their trained parameters using an online conversational interface. AI has opened new avenues in redefining the role boundaries of teachers and learners and has the potential to impact the teaching-learning process.

In this descriptive proof-of- concept cross-sectional study we have explored the application of three generative AI tools on drug treatment of hypertension theme to generate: (1) specific learning outcomes (SLOs); (2) test items (MCQs- A type and case cluster; SAQs; OSPE); (3) test standard-setting parameters for medical students.

Analysis of AI-generated output showed profound homology but divergence in quality and responsiveness to refining search queries. The SLOs identified key domains of antihypertensive pharmacology and therapeutics relevant to stages of the medical program, stated with appropriate action verbs as per Bloom’s taxonomy. Test items often had clinical vignettes aligned with the key domain stated in search queries. Some test items related to A-type MCQs had construction defects, multiple correct answers, and dubious appropriateness to the learner’s stage. ChatGPT generated explanations for test items, this enhancing usefulness to support self-study by learners. Integrated case-cluster items had focused clinical case description vignettes, integration across disciplines, and targeted higher levels of competencies. The response of AI tools on standard-setting varied. Individual questions for each SAQ clinical scenario were mostly open-ended. The AI-generated OSPE test items were appropriate for the learner’s stage and identified relevant pharmacotherapeutic issues. The model answers supplied for both SAQs and OSPEs can aid course instructors in planning classroom lessons, identifying suitable instructional methods, establishing rubrics for grading, and for learners as a study guide. Key lessons learnt for improving AI-generated test item quality are outlined.

Conclusions

AI tools are useful adjuncts to plan instructional methods, identify themes for test blueprinting, generate test items, and guide test standard-setting appropriate to learners’ stage in the medical program. However, experts need to review the content validity of AI-generated output. We expect AIs to influence the medical education landscape to empower learners, and to align competencies with curriculum implementation. AI literacy is an essential competency for health professionals.

Peer Review reports

Artificial intelligence (AI) has great potential to revolutionize the field of medical education from curricular conception to assessment [ 1 ]. AIs used in medical education are mostly generative AI large language models that were developed and validated based on billions to trillions of parameters [ 2 ]. AIs hold promise in the incorporation of history-taking, assessment, diagnosis, and management of various disorders [ 3 ]. While applications of AIs in undergraduate medical training are being explored, huge ethical challenges remain in terms of data collection, maintaining anonymity, consent, and ownership of the provided data [ 4 ]. AIs hold a promising role amongst learners because they can deliver a personalized learning experience by tracking their progress and providing real-time feedback, thereby enhancing their understanding in the areas they are finding difficult [ 5 ]. Consequently, a recent survey has shown that medical students have expressed their interest in acquiring competencies related to the use of AIs in healthcare during their undergraduate medical training [ 6 ].

Pharmacology and Therapeutics (P & T) is a core discipline embedded in the undergraduate medical curriculum, mostly in the pre-clerkship phase. However, the application of therapeutic principles forms one of the key learning objectives during the clerkship phase of the undergraduate medical career. Student assessment in pharmacology & therapeutics (P&T) is with test items such as multiple-choice questions (MCQs), integrated case cluster questions, short answer questions (SAQs), and objective structured practical examination (OSPE) in the undergraduate medical curriculum. It has been argued that AIs possess the ability to communicate an idea more creatively than humans [ 7 ]. It is imperative that with access to billions of trillions of datasets the AI platforms hold promise in playing a crucial role in the conception of various test items related to any of the disciplines in the undergraduate medical curriculum. Additionally, AIs provide an optimized curriculum for a program/course/topic addressing multidimensional problems [ 8 ], although robust evidence for this claim is lacking.

The existing literature has evaluated the knowledge, attitude, and perceptions of adopting AI in medical education. Integration of AIs in medical education is the need of the hour in all health professional education. However, the academic medical fraternity facing challenges in the incorporation of AIs in the medical curriculum due to factors such as inadequate grounding in data analytics, lack of high-quality firm evidence favoring the utility of AIs in medical education, and lack of funding [ 9 ]. Open-access AI platforms are available free to users without any restrictions. Hence, as a proof-of-concept, we chose to explore the utility of three AI platforms to identify specific learning objectives (SLOs) related to pharmacology discipline in the management of hypertension for medical students at different stages of their medical training.

Study design and ethics

The present study is observational, cross-sectional in design, conducted in the Department of Pharmacology & Therapeutics, College of Medicine and Medical Sciences, Arabian Gulf University, Kingdom of Bahrain, between April and August 2023. Ethical Committee approval was not sought given the nature of this study that neither had any interaction with humans, nor collection of any personal data was involved.

Study procedure

We conducted the present study in May-June 2023 with the Poe© chatbot interface created by Quora© that provides access to the following three AI platforms:

Sage Poe [ 10 ]: A generative AI search engine developed by Anthropic © that conceives a response based on the written input provided. Quora has renamed Sage Poe as Assistant © from July 2023 onwards.

Claude-Instant [ 11 ]: A retrieval-based AI search engine developed by Anthropic © that collates a response based on pre-written responses amongst the existing databases.

ChatGPT version 3.5 [ 12 ]: A generative architecture-based AI search engine developed by OpenAI © trained on large and diverse datasets.

We queried the chatbots to generate SLOs, A-type MCQs, integrated case cluster MCQs, integrated SAQs, and OSPE test items in the domain of systemic hypertension related to the P&T discipline. Separate prompts were used to generate outputs for pre-clerkship (preclinical) phase students, and at the time of graduation (before starting residency programs). Additionally, we have also evaluated the ability of these AI platforms to estimate the proportion of students correctly answering these test items. We used the following queries for each of these objectives:

Specific learning objectives

Can you generate specific learning objectives in the pharmacology discipline relevant to undergraduate medical students during their pre-clerkship phase related to anti-hypertensive drugs?

Can you generate specific learning objectives in the pharmacology discipline relevant to undergraduate medical students at the time of graduation related to anti-hypertensive drugs?

A-type MCQs

In the initial query used for A-type of item, we specified the domains (such as the mechanism of action, pharmacokinetics, adverse reactions, and indications) so that a sample of test items generated without any theme-related clutter, shown below:

Write 20 single best answer MCQs with 5 choices related to anti-hypertensive drugs for undergraduate medical students during the pre-clerkship phase of which 5 MCQs should be related to mechanism of action, 5 MCQs related to pharmacokinetics, 5 MCQs related to adverse reactions, and 5 MCQs should be related to indications.

The MCQs generated with the above search query were not based on clinical vignettes. We queried again to generate MCQs using clinical vignettes specifically because most medical schools have adopted problem-based learning (PBL) in their medical curriculum.

Write 20 single best answer MCQs with 5 choices related to anti-hypertensive drugs for undergraduate medical students during the pre-clerkship phase using a clinical vignette for each MCQ of which 5 MCQs should be related to the mechanism of action, 5 MCQs related to pharmacokinetics, 5 MCQs related to adverse reactions, and 5 MCQs should be related to indications.

We attempted to explore whether AI platforms can provide useful guidance on standard-setting. Hence, we used the following search query.

Can you do a simulation with 100 undergraduate medical students to take the above questions and let me know what percentage of students got each MCQ correct?

Integrated case cluster MCQs

Write 20 integrated case cluster MCQs with 2 questions in each cluster with 5 choices for undergraduate medical students during the pre-clerkship phase integrating pharmacology and physiology related to systemic hypertension with a case vignette.

Write 20 integrated case cluster MCQs with 2 questions in each cluster with 5 choices for undergraduate medical students during the pre-clerkship phase integrating pharmacology and physiology related to systemic hypertension with a case vignette. Please do not include ‘none of the above’ as the choice. (This modified search query was used because test items with ‘None of the above’ option were generated with the previous search query).

Write 20 integrated case cluster MCQs with 2 questions in each cluster with 5 choices for undergraduate medical students at the time of graduation integrating pharmacology and physiology related to systemic hypertension with a case vignette.

Integrated short answer questions

Write a short answer question scenario with difficult questions based on the theme of a newly diagnosed hypertensive patient for undergraduate medical students with the main objectives related to the physiology of blood pressure regulation, risk factors for systemic hypertension, pathophysiology of systemic hypertension, pathological changes in the systemic blood vessels in hypertension, pharmacological management, and non-pharmacological treatment of systemic hypertension.

Write a short answer question scenario with moderately difficult questions based on the theme of a newly diagnosed hypertensive patient for undergraduate medical students with the main objectives related to the physiology of blood pressure regulation, risk factors for systemic hypertension, pathophysiology of systemic hypertension, pathological changes in the systemic blood vessels in hypertension, pharmacological management, and non-pharmacological treatment of systemic hypertension.

Write a short answer question scenario with questions based on the theme of a newly diagnosed hypertensive patient for undergraduate medical students at the time of graduation with the main objectives related to the physiology of blood pressure regulation, risk factors for systemic hypertension, pathophysiology of systemic hypertension, pathological changes in the systemic blood vessels in hypertension, pharmacological management, and non-pharmacological treatment of systemic hypertension.

Can you generate 5 OSPE pharmacology and therapeutics prescription writing exercises for the assessment of undergraduate medical students at the time of graduation related to anti-hypertensive drugs?

Can you generate 5 OSPE pharmacology and therapeutics prescription writing exercises containing appropriate instructions for the patients for the assessment of undergraduate medical students during their pre-clerkship phase related to anti-hypertensive drugs?

Can you generate 5 OSPE pharmacology and therapeutics prescription writing exercises containing appropriate instructions for the patients for the assessment of undergraduate medical students at the time of graduation related to anti-hypertensive drugs?

Both authors independently evaluated the AI-generated outputs, and a consensus was reached. We cross-checked the veracity of answers suggested by AIs as per the Joint National Commission Guidelines (JNC-8) and Goodman and Gilman’s The Pharmacological Basis of Therapeutics (2023), a reference textbook [ 13 , 14 ]. Errors in the A-type MCQs were categorized as item construction defects, multiple correct answers, and uncertain appropriateness to the learner’s level. Test items in the integrated case cluster MCQs, SAQs and OSPEs were evaluated with the Preliminary Conceptual Framework for Establishing Content Validity of AI-Generated Test Items based on the following domains: technical accuracy, comprehensiveness, education level, and lack of construction defects (Table  1 ). The responses were categorized as complete and deficient for each domain.

The pre-clerkship phase SLOs identified by Sage Poe, Claude-Instant, and ChatGPT are listed in the electronic supplementary materials 1 – 3 , respectively. In general, a broad homology in SLOs generated by the three AI platforms was observed. All AI platforms identified appropriate action verbs as per Bloom’s taxonomy to state the SLO; action verbs such as describe, explain, recognize, discuss, identify, recommend, and interpret are used to state the learning outcome. The specific, measurable, achievable, relevant, time-bound (SMART) SLOs generated by each AI platform slightly varied. All key domains of antihypertensive pharmacology to be achieved during the pre-clerkship (pre-clinical) years were relevant for graduating doctors. The SLOs addressed current JNC Treatment Guidelines recommended classes of antihypertensive drugs, the mechanism of action, pharmacokinetics, adverse effects, indications/contraindications, dosage adjustments, monitoring therapy, and principles of monotherapy and combination therapy.

The SLOs to be achieved by undergraduate medical students at the time of graduation identified by Sage Poe, Claude-Instant, and ChatGPT listed in electronic supplementary materials 4 – 6 , respectively. The identified SLOs emphasize the application of pharmacology knowledge within a clinical context, focusing on competencies needed to function independently in early residency stages. These SLOs go beyond knowledge recall and mechanisms of action to encompass competencies related to clinical problem-solving, rational prescribing, and holistic patient management. The SLOs generated require higher cognitive ability of the learner: action verbs such as demonstrate, apply, evaluate, analyze, develop, justify, recommend, interpret, manage, adjust, educate, refer, design, initiate & titrate were frequently used.

The MCQs for the pre-clerkship phase identified by Sage Poe, Claude-Instant, and ChatGPT listed in the electronic supplementary materials 7 – 9 , respectively, and those identified with the search query based on the clinical vignette in electronic supplementary materials ( 10 – 12 ).

All MCQs generated by the AIs in each of the four domains specified [mechanism of action (MOA); pharmacokinetics; adverse drug reactions (ADRs), and indications for antihypertensive drugs] are quality test items with potential content validity. The test items on MOA generated by Sage Poe included themes such as renin-angiotensin-aldosterone (RAAS) system, beta-adrenergic blockers (BB), calcium channel blockers (CCB), potassium channel openers, and centrally acting antihypertensives; on pharmacokinetics included high oral bioavailability/metabolism in liver [angiotensin receptor blocker (ARB)-losartan], long half-life and renal elimination [angiotensin converting enzyme inhibitors (ACEI)-lisinopril], metabolism by both liver and kidney (beta-blocker (BB)-metoprolol], rapid onset- short duration of action (direct vasodilator-hydralazine), and long-acting transdermal drug delivery (centrally acting-clonidine). Regarding the ADR theme, dry cough, angioedema, and hyperkalemia by ACEIs in susceptible patients, reflex tachycardia by CCB/amlodipine, and orthostatic hypotension by CCB/verapamil addressed. Clinical indications included the drug of choice for hypertensive patients with concomitant comorbidity such as diabetics (ACEI-lisinopril), heart failure and low ejection fraction (BB-carvedilol), hypertensive urgency/emergency (alpha cum beta receptor blocker-labetalol), stroke in patients with history recurrent stroke or transient ischemic attack (ARB-losartan), and preeclampsia (methyldopa).

Almost similar themes under each domain were identified by the Claude-Instant AI platform with few notable exceptions: hydrochlorothiazide (instead of clonidine) in MOA and pharmacokinetics domains, respectively; under the ADR domain ankle edema/ amlodipine, sexual dysfunction and fatigue in male due to alpha-1 receptor blocker; under clinical indications the best initial monotherapy for clinical scenarios such as a 55-year old male with Stage-2 hypertension; a 75-year-old man Stage 1 hypertension; a 35-year-old man with Stage I hypertension working on night shifts; and a 40-year-old man with stage 1 hypertension and hyperlipidemia.

As with Claude-Instant AI, ChatGPT-generated test items on MOA were mostly similar. However, under the pharmacokinetic domain, immediate- and extended-release metoprolol, the effect of food to enhance the oral bioavailability of ramipril, and the highest oral bioavailability of amlodipine compared to other commonly used antihypertensives were the themes identified. Whereas the other ADR themes remained similar, constipation due to verapamil was a new theme addressed. Notably, in this test item, amlodipine was an option that increased the difficulty of this test item because amlodipine therapy is also associated with constipation, albeit to a lesser extent, compared to verapamil. In the clinical indication domain, the case description asking “most commonly used in the treatment of hypertension and heart failure” is controversial because the options listed included losartan, ramipril, and hydrochlorothiazide but the suggested correct answer was ramipril. This is a good example to stress the importance of vetting the AI-generated MCQ by experts for content validity and to assure robust psychometrics. The MCQ on the most used drug in the treatment of “hypertension and diabetic nephropathy” is more explicit as opposed to “hypertension and diabetes” by Claude-Instant because the therapeutic concept of reducing or delaying nephropathy must be distinguished from prevention of nephropathy, although either an ACEI or ARB is the drug of choice for both indications.

It is important to align student assessment to the curriculum; in the PBL curriculum, MCQs with a clinical vignette are preferred. The modification of the query specifying the search to generate MCQs with a clinical vignette on domains specified previously gave appropriate output by all three AI platforms evaluated (Sage Poe; Claude- Instant; Chat GPT). The scenarios generated had a good clinical fidelity and educational fit for the pre-clerkship student perspective.

The errors observed with AI outputs on the A-type MCQs are summarized in Table  2 . No significant pattern was observed except that Claude-Instant© generated test items in a stereotyped format such as the same choices for all test items related to pharmacokinetics and indications, and all the test items in the ADR domain are linked to the mechanisms of action of drugs. This illustrates the importance of reviewing AI-generated test items by content experts for content validity to ensure alignment with evidence-based medicine and up-to-date treatment guidelines.

The test items generated by ChatGPT had the advantage of explanations supplied rendering these more useful for learners to support self-study. The following examples illustrate this assertion: “ A patient with hypertension is started on a medication that works by blocking beta-1 receptors in the heart (metoprolol)”. Metoprolol is a beta blocker that works by blocking beta-1 receptors in the heart, which reduces heart rate and cardiac output, resulting in a decrease in blood pressure. However, this explanation is incomplete because there is no mention of other less important mechanisms, of beta receptor blockers on renin release. Also, these MCQs were mostly recall type: Which of the following medications is known to have a significant first-pass effect? The explanation reads: propranolol is known to have a significant first pass-effect, meaning that a large portion of the drug is metabolized by the liver before it reaches systemic circulation. Losartan, amlodipine, ramipril, and hydrochlorothiazide do not have significant first-pass effect. However, it is also important to extend the explanation further by stating that the first-pass effect of propranolol does not lead to total loss of pharmacological activity because the metabolite hydroxy propranolol also has potent beta-blocking activity. Another MCQ test item had a construction defect: “A patient with hypertension is started on a medication that can cause photosensitivity. Which of the following medications is most likely responsible?” Options included: losartan, amlodipine, ramipril, hydrochlorothiazide, hydrochlorothiazide/triamterene. The explanation states that hydrochlorothiazide (HCTZ) can cause photosensitivity, which is increased sensitivity to sunlight that can cause skin rashes and sunburns. Losartan, amlodipine, ramipril, and HCTZ/triamterene are less likely to cause photosensitivity. However, it is irrational to claim that HCTZ/triamterene, one of the wrong options, is less likely to cause photosensitivity. The difficulty index of such test items is likely unacceptable in high-stake tests to make equitable psychometric decisions.

The responses from Sage Poe, Claude-Instant, and ChatGPT on guidance for standard-setting methods are listed in the electronic supplementary materials 13 – 15 . Sage Poe offered this information with the caveat that “the results of this simulation were purely hypothetical and do not reflect the actual performance of any real students”. The simulation estimates of examinees correctly answering test items ranged from 60 to 90% for the mechanism of action, 65–85% for pharmacokinetics, 60–80% for adverse reactions, and 75–90% for clinical indications of antihypertensives.

In contrast, Claude-Instant admitted that “Performing a full simulation with 100 medical students is beyond the scope of what I can do as an AI assistant. I can generate sample MCQ questions and answers, but I do not have access to a population of medical students to administer the questions. A true simulation that administers questions to real students and analyzes the results would require access to a database of medical students, a platform to deliver the questions, and code to analyze the responses. As an AI assistant, I lack these capabilities. While I cannot perform a full simulation, here are some considerations for administering MCQ questions to medical students: provide details on the purpose and instructions for the test to ensure students understand what is expected out of them; consider including some easier questions in addition to more challenging ones to boost students’ confidence and motivation; avoid ambiguous wording and ensure each question has only one unambiguously correct answer option; when analyzing results, look for questions factors that may have impacted performance, like unclear wording or answer options that are too similar. Focusing on weak questions can help improve future tests; compare performance across different student groups to identify areas where more teaching may be needed; provide detailed answer explanations and review sessions after the test to help students learn from any errors and reinforce correct concepts”. However, ChatGPT offered the simulation result estimates for the 20 MCQ on antihypertensive drugs: mechanism of action ranged from 67 to 92%, pharmacokinetics 63–86%, adverse effects 65–82%, and clinical indications 64–89%. Furthermore, it also stated that “Overall, the performance of the students was quite good, with most questions having a response rate of over 70%. However, there were some questions where the projected correct response rate was rather low, such as Question #5 (Mechanism of Action of Hydrochlorothiazide; see Electronic Supplementary Material 12 ) and Question 18 (Indications for Verapamil; see Electronic Supplementary Material 10 ). This may suggest areas where students need more focused education or review.”

We asked AI assistants to generate 20 integrated case cluster MCQs with 2 test items in each cluster with five options for undergraduate medical students in the pre-clerkship phase integrating pharmacology and physiology related to systemic hypertension with a case vignette and the responses by Sage Poe, Claude-Instant, and ChatGPT are listed in the electronic supplementary materials ( 16 – 18 ). In all instances, the test items generated had focused case descriptions in the form of a clinical vignette, and horizontal integration across the pathophysiology of hypertension and pharmacology of antihypertensive drugs. These test items mostly targeted the ‘knows (knowledge)’ or ‘knows how (competence)’ level on Miller’s pyramid and are suitable for assessing the clinical competence of pre-clerkship medical students, especially in an integrated PBL curriculum. Both the AI assistants generated excellent clinical vignettes and themes; however, most of the cluster MCQs by ChatGPT had “None of the above” as an option, which is often considered a test item construction flaw. Notwithstanding these limitations, case cluster integrated test items are valuable for learners to integrate their knowledge of different basic medical sciences and their application to clinical sciences. This integrated approach can be used for both instructional and student assessment purposes to make the course more meaningful. Indeed, one of the basic tenets of PBL is curriculum integration.

In the next query, we asked AI assistants to write integrated case cluster MCQs with 2 test items in each cluster with 5 options for undergraduate medical students at the time of graduation integrating pharmacology and physiology of hypertension with a case vignette. The responses are listed in the electronic supplementary materials 19 – 21 . Claude-Instant has provided only partial responses to both the queries. Overall, the information provided in case vignettes requires the learner to interpret the clinical information and make therapeutic decisions. Such test items are appropriate for assessing the minimum competency of medical school graduates to start internship or residency programs; these MCQs are generally comparable to test items used in USMLE Part 2. The evaluation of the integrated case cluster test items provided by all the AI platforms is summarized in Table  3 . In general, the AI outputs had at least one construction defect, and did not include all the core sub-topics to a sufficient degree.

When asked to simulate the above-generated test items to 100 undergraduate medical students at the time of graduation and provide a percentage of students who are likely to correctly answer each item, the AI assistant responded:

“ As an AI language model, I cannot simulate the responses of 100 undergraduate medical students likely to obtain the correct answer for each question. However, I can provide some general information about the difficulty level of each question based on the complexity of the concepts being tested and the level of integration between pharmacology and physiology ”.

AI assistant went on to clarify that questions that require knowledge of basic pharmacology principles, such as the mechanism of action of specific drugs, are likely to be easier for students to answer correctly. Test items that require an understanding of the physiological mechanisms underlying hypertension and correlating with symptoms are likely to be more challenging for students. The AI assistant sorted these test items into two categories accordingly. Overall, the difficulty level of the test item is based on the level of integration between pharmacology and pathophysiology. Test items that require an understanding of both pharmacological and physiological mechanisms are likely to be more challenging for students requiring a strong foundation in both pharmacology and physiology concepts to be able to correctly answer integrated case-cluster MCQs.

Short answer questions

The responses to a search query on generating SAQs appropriate to the pre-clerkship phase Sage Poe, Claude-Instant, and ChatGPT generated items are listed in the electronic supplementary materials 22 – 24 for difficult questions and 25–27 for moderately difficult questions.

It is apparent from these case vignette descriptions that the short answer question format varied. Accordingly, the scope for asking individual questions for each scenario is open-ended. In all instances, model answers are supplied which are helpful for the course instructor to plan classroom lessons, identify appropriate instructional methods, and establish rubrics for grading the answer scripts, and as a study guide for students.

We then wanted to see to what extent AI can differentiate the difficulty of the SAQ by replacing the search term “difficult” with “moderately difficult” in the above search prompt: the changes in the revised case scenarios are substantial. Perhaps the context of learning and practice (and the level of the student in the MD/medical program) may determine the difficulty level of SAQ generated. It is worth noting that on changing the search from cardiology to internal medicine rotation in Sage Poe the case description also changed. Thus, it is essential to select an appropriate AI assistant, perhaps by trial and error, to generate quality SAQs. Most of the individual questions tested stand-alone knowledge and did not require students to demonstrate integration.

The responses of Sage Poe, Claude-Instant, and ChatGPT for the search query to generate SAQs at the time of graduation are listed in the electronic supplementary materials 28 – 30 . It is interesting to note how AI assistants considered the stage of the learner while generating the SAQ. The response by Sage Poe is illustrative for comparison. “You are a newly graduated medical student who is working in a hospital” versus “You are a medical student in your pre-clerkship.”

Some questions were retained, deleted, or modified to align with competency appropriate to the context (Electronic Supplementary Materials 28 – 30 ). Overall, the test items at both levels from all AI platforms were technically accurate and thorough addressing the topics related to different disciplines (Table  3 ). The differences in learning objective transition are summarized in Table  4 . A comparison of learning objectives revealed that almost all objectives remained the same except for a few (Table  5 ).

A similar trend was apparent with test items generated by other AI assistants, such as ChatGPT. The contrasting differences in questions are illustrated by the vertical integration of basic sciences and clinical sciences (Table  6 ).

Taken together, these in-depth qualitative comparisons suggest that AI assistants such as Sage Poe and ChatGPT consider the learner’s stage of training in designing test items, learning outcomes, and answers expected from the examinee. It is critical to state the search query explicitly to generate quality output by AI assistants.

The OSPE test items generated by Claude-Instant and ChatGPT appropriate to the pre-clerkship phase (without mentioning “appropriate instructions for the patients”) are listed in the electronic supplementary materials 31 and 32 and with patient instructions on the electronic supplementary materials 33 and 34 . For reasons unknown, Sage Poe did not provide any response to this search query.

The five OSPE items generated were suitable to assess the prescription writing competency of pre-clerkship medical students. The clinical scenarios identified by the three AI platforms were comparable; these scenarios include patients with hypertension and impaired glucose tolerance in a 65-year-old male, hypertension with chronic kidney disease (CKD) in a 55-year-old woman, resistant hypertension with obstructive sleep apnea in a 45-year-old man, and gestational hypertension at 32 weeks in a 35-year-old (Claude-Instant AI). Incorporating appropriate instructions facilitates the learner’s ability to educate patients and maximize safe and effective therapy. The OSPE item required students to write a prescription with guidance to start conservatively, choose an appropriate antihypertensive drug class (drug) based on the patients’ profile, specifying drug name, dose, dosing frequency, drug quantity to be dispensed, patient name, date, refill, and caution as appropriate, in addition to prescribers’ name, signature, and license number. In contrast, ChatGPT identified clinical scenarios to include patients with hypertension and CKD, hypertension and bronchial asthma, gestational diabetes, hypertension and heart failure, and hypertension and gout (ChatGPT). Guidance for dosage titration, warnings to be aware, safety monitoring, and frequency of follow-up and dose adjustment. These test items are designed to assess learners’ knowledge of P & T of antihypertensives, as well as their ability to provide appropriate instructions to patients. These clinical scenarios for writing prescriptions assess students’ ability to choose an appropriate drug class, write prescriptions with proper labeling and dosing, reflect drug safety profiles, and risk factors, and make modifications to meet the requirements of special populations. The prescription is required to state the drug name, dose, dosing frequency, patient name, date, refills, and cautions or instructions as needed. A conservative starting dose, once or twice daily dosing frequency based on the drug, and instructions to titrate the dose slowly if required.

The responses from Claude-Instant and ChatGPT for the search query related to generating OSPE test items at the time of graduation are listed in electronic supplementary materials 35 and 36 . In contrast to the pre-clerkship phase, OSPEs generated for graduating doctors’ competence assessed more advanced drug therapy comprehension. For example, writing a prescription for:

(1) A 65-year- old male with resistant hypertension and CKD stage 3 to optimize antihypertensive regimen required the answer to include starting ACEI and diuretic, titrating the dosage over two weeks, considering adding spironolactone or substituting ACEI with an ARB, and need to closely monitor serum electrolytes and kidney function closely.

(2) A 55-year-old woman with hypertension and paroxysmal arrhythmia required the answer to include switching ACEI to ARB due to cough, adding a CCB or beta blocker for rate control needs, and adjusting the dosage slowly and monitoring for side effects.

(3) A 45-year-old man with masked hypertension and obstructive sleep apnea require adding a centrally acting antihypertensive at bedtime and increasing dosage as needed based on home blood pressure monitoring and refer to CPAP if not already using one.

(4) A 75-year-old woman with isolated systolic hypertension and autonomic dysfunction to require stopping diuretic and switching to an alpha blocker, upward dosage adjustment and combining with other antihypertensives as needed based on postural blood pressure changes and symptoms.

(5) A 35-year-old pregnant woman with preeclampsia at 29 weeks require doubling methyldopa dose and consider adding labetalol or nifedipine based on severity and educate on signs of worsening and to follow-up immediately for any concerning symptoms.

These case scenarios are designed to assess the ability of the learner to comprehend the complexity of antihypertensive regimens, make evidence-based regimen adjustments, prescribe multidrug combinations based on therapeutic response and tolerability, monitor complex patients for complications, and educate patients about warning signs and follow-up.

A similar output was provided by ChatGPT, with clinical scenarios such as prescribing for patients with hypertension and myocardial infarction; hypertension and chronic obstructive pulmonary airway disease (COPD); hypertension and a history of angina; hypertension and a history of stroke, and hypertension and advanced renal failure. In these cases, wherever appropriate, pharmacotherapeutic issues like taking ramipril after food to reduce side effects such as giddiness; selection of the most appropriate beta-blocker such as nebivolol in patients with COPD comorbidity; the importance of taking amlodipine at the same time every day with or without food; preference for telmisartan among other ARBs in stroke; choosing furosemide in patients with hypertension and edema and taking the medication with food to reduce the risk of gastrointestinal adverse effect are stressed.

The AI outputs on OSPE test times were observed to be technically accurate, thorough in addressing core sub-topics suitable for the learner’s level and did not have any construction defects (Table  3 ). Both AIs provided the model answers with explanatory notes. This facilitates the use of such OSPEs for self-assessment by learners for formative assessment purposes. The detailed instructions are helpful in creating optimized therapy regimens, and designing evidence-based regimens, to provide appropriate instructions to patients with complex medical histories. One can rely on multiple AI sources to identify, shortlist required case scenarios, and OSPE items, and seek guidance on expected model answers with explanations. The model answer guidance for antihypertensive drug classes is more appropriate (rather than a specific drug of a given class) from a teaching/learning perspective. We believe that these scenarios can be refined further by providing a focused case history along with relevant clinical and laboratory data to enhance clinical fidelity and bring a closer fit to the competency framework.

In the present study, AI tools have generated SLOs that comply with the current principles of medical education [ 15 ]. AI tools are valuable in constructing SLOs and so are especially useful for medical fraternities where training in medical education is perceived as inadequate, more so in the early stages of their academic career. Data suggests that only a third of academics in medical schools have formal training in medical education [ 16 ] which is a limitation. Thus, the credibility of alternatives, such as the AIs, is evaluated to generate appropriate course learning outcomes.

We observed that the AI platforms in the present study generated quality test items suitable for different types of assessment purposes. The AI-generated outputs were similar with minor variation. We have used generative AIs in the present study that could generate new content from their training dataset [ 17 ]. Problem-based and interactive learning approaches are referred to as “bottom-up” where learners obtain first-hand experience in solving the cases first and then indulge in discussion with the educators to refine their understanding and critical thinking skills [ 18 ]. We suggest that AI tools can be useful for this approach for imparting the core knowledge and skills related to Pharmacology and Therapeutics to undergraduate medical students. A recent scoping review evaluating the barriers to writing quality test items based on 13 studies has concluded that motivation, time constraints, and scheduling were the most common [ 19 ]. AI tools can be valuable considering the quick generation of quality test items and time management. However, as observed in the present study, the AI-generated test items nevertheless require scrutiny by faculty members for content validity. Moreover, it is important to train faculty in AI technology-assisted teaching and learning. The General Medical Council recommends taking every opportunity to raise the profile of teaching in medical schools [ 20 ]. Hence, both the academic faculty and the institution must consider investing resources in AI training to ensure appropriate use of the technology [ 21 ].

The AI outputs assessed in the present study had errors, particularly with A-type MCQs. One notable observation was that often the AI tools were unable to differentiate the differences between ACEIs and ARBs. AI platforms access several structured and unstructured data, in addition to images, audio, and videos. Hence, the AI platforms can commit errors due to extracting details from unauthenticated sources [ 22 ] created a framework identifying 28 factors for reconstructing the path of AI failures and for determining corrective actions. This is an area of interest for AI technical experts to explore. Also, this further iterates the need for human examination of test items before using them for assessment purposes.

There are concerns that AIs can memorize and provide answers from their training dataset, which they are not supposed to do [ 23 ]. Hence, the use of AIs-generated test items for summative examinations is debatable. It is essential to ensure and enhance the security features of AI tools to reduce or eliminate cross-contamination of test items. Researchers have emphasized that AI tools will only reach their potential if developers and users can access full-text non-PDF formats that help machines comprehend research papers and generate the output [ 24 ].

AI platforms may not always have access to all standard treatment guidelines. However, in the present study, it was observed that all three AI platforms generally provided appropriate test items regarding the choice of medications, aligning with recommendations from contemporary guidelines and standard textbooks in pharmacology and therapeutics. The prompts used in the study were specifically focused on the pre-clerkship phase of the undergraduate medical curriculum (and at the time of their graduation) and assessed fundamental core concepts, which were also reflected in the AI outputs. Additionally, the recommended first-line antihypertensive drug classes have been established for several decades, and information regarding their pharmacokinetics, ADRs, and indications is well-documented in the literature.

Different paradigms and learning theories have been proposed to support AI in education. These paradigms include AI- directed (learner as recipient), AI-supported (learner as collaborator), and AI-empowered (learner as leader) that are based on Behaviorism, Cognitive-Social constructivism, and Connectivism-Complex adaptive systems, respectively [ 25 ]. AI techniques have potential to stimulate and advance instructional and learning sciences. More recently a three- level model that synthesizes and unifies existing learning theories to model the roles of AIs in promoting learning process has been proposed [ 26 ]. The different components of our study rely upon these paradigms and learning theories as the theoretical underpinning.

Strengths and limitations

To the best of our knowledge, this is the first study evaluating the utility of AI platforms in generating test items related to a discipline in the undergraduate medical curriculum. We have evaluated the AI’s ability to generate outputs related to most types of assessment in the undergraduate medical curriculum. The key lessons learnt for improving the AI-generated test item quality from the present study are outlined in Table  7 . We used a structured framework for assessing the content validity of the test items. However, we have demonstrated using a single case study (hypertension) as a pilot experiment. We chose to evaluate anti-hypertensive drugs as it is a core learning objective and one of the most common disorders relevant to undergraduate medical curricula worldwide. It would be interesting to explore the output from AI platforms for other common (and uncommon/region-specific) disorders, non-/semi-core objectives, and disciplines other than Pharmacology and Therapeutics. An area of interest would be to look at the content validity of the test items generated for different curricula (such as problem-based, integrated, case-based, and competency-based) during different stages of the learning process. Also, we did not attempt to evaluate the generation of flowcharts, algorithms, or figures for generating test items. Another potential area for exploring the utility of AIs in medical education would be repeated procedural practices such as the administration of drugs through different routes by trainee residents [ 27 ]. Several AI tools have been identified for potential application in enhancing classroom instructions and assessment purposes pending validation in prospective studies [ 28 ]. Lastly, we did not administer the AI-generated test items to students and assessed their performance and so could not comment on the validity of test item discrimination and difficulty indices. Additionally, there is a need to confirm the generalizability of the findings to other complex areas in the same discipline as well as in other disciplines that pave way for future studies. The conceptual framework used in the present study for evaluating the AI-generated test items needs to be validated in a larger population. Future studies may also try to evaluate the variations in the AI outputs with repetition of the same queries.

Notwithstanding ongoing discussions and controversies, AI tools are potentially useful adjuncts to optimize instructional methods, test blueprinting, test item generation, and guidance for test standard-setting appropriate to learners’ stage in the medical program. However, experts need to critically review the content validity of AI-generated output. These challenges and caveats are to be addressed before the use of widespread use of AIs in medical education can be advocated.

Data availability

All the data included in this study are provided as Electronic Supplementary Materials.

Tolsgaard MG, Pusic MV, Sebok-Syer SS, Gin B, Svendsen MB, Syer MD, Brydges R, Cuddy MM, Boscardin CK. The fundamentals of Artificial Intelligence in medical education research: AMEE Guide 156. Med Teach. 2023;45(6):565–73.

Article   Google Scholar  

Sriwastwa A, Ravi P, Emmert A, Chokshi S, Kondor S, Dhal K, Patel P, Chepelev LL, Rybicki FJ, Gupta R. Generative AI for medical 3D printing: a comparison of ChatGPT outputs to reference standard education. 3D Print Med. 2023;9(1):21.

Azer SA, Guerrero APS. The challenges imposed by artificial intelligence: are we ready in medical education? BMC Med Educ. 2023;23(1):680.

Masters K. Ethical use of Artificial Intelligence in Health Professions Education: AMEE Guide 158. Med Teach. 2023;45(6):574–84.

Nagi F, Salih R, Alzubaidi M, Shah H, Alam T, Shah Z, Househ M. Applications of Artificial Intelligence (AI) in Medical Education: a scoping review. Stud Health Technol Inf. 2023;305:648–51.

Google Scholar  

Mehta N, Harish V, Bilimoria K, et al. Knowledge and attitudes on artificial intelligence in healthcare: a provincial survey study of medical students. MedEdPublish. 2021;10(1):75.

Mir MM, Mir GM, Raina NT, Mir SM, Mir SM, Miskeen E, Alharthi MH, Alamri MMS. Application of Artificial Intelligence in Medical Education: current scenario and future perspectives. J Adv Med Educ Prof. 2023;11(3):133–40.

Garg T. Artificial Intelligence in Medical Education. Am J Med. 2020;133(2):e68.

Matheny ME, Whicher D, Thadaney IS. Artificial intelligence in health care: a report from the National Academy of Medicine. JAMA. 2020;323(6):509–10.

Sage Poe. Available at: https://poe.com/Assistant (Accessed on. 3rd June 2023).

Claude-Instant: Available at: https://poe.com/Claude-instant (Accessed on 3rd. June 2023).

ChatGPT: Available at: https://poe.com/ChatGPT (Accessed on 3rd. June 2023).

James PA, Oparil S, Carter BL, Cushman WC, Dennison-Himmelfarb C, Handler J, Lackland DT, LeFevre ML, MacKenzie TD, Ogedegbe O, Smith SC Jr, Svetkey LP, Taler SJ, Townsend RR, Wright JT Jr, Narva AS, Ortiz E. 2014 evidence-based guideline for the management of high blood pressure in adults: report from the panel members appointed to the Eighth Joint National Committee (JNC 8). JAMA. 2014;311(5):507–20.

Eschenhagen T. Treatment of hypertension. In: Brunton LL, Knollmann BC, editors. Goodman & Gilman’s the pharmacological basis of therapeutics. 14th ed. New York: McGraw Hill; 2023.

Shabatura J. September. Using Bloom’s taxonomy to write effective learning outcomes. https://tips.uark.edu/using-blooms-taxonomy/ (Accessed on 19th 2023).

Trainor A, Richards JB. Training medical educators to teach: bridging the gap between perception and reality. Isr J Health Policy Res. 2021;10(1):75.

Boscardin C, Gin B, Golde PB, Hauer KE. ChatGPT and generative artificial intelligence for medical education: potential and opportunity. Acad Med. 2023. https://doi.org/10.1097/ACM.0000000000005439 . (Published ahead of print).

Duong MT, Rauschecker AM, Rudie JD, Chen PH, Cook TS, Bryan RN, Mohan S. Artificial intelligence for precision education in radiology. Br J Radiol. 2019;92(1103):20190389.

Karthikeyan S, O’Connor E, Hu W. Barriers and facilitators to writing quality items for medical school assessments - a scoping review. BMC Med Educ. 2019;19(1):123.

Developing teachers and trainers in undergraduate medical education. Advice supplementary to Tomorrow’s Doctors. (2009). https://www.gmc-uk.org/-/media/documents/Developing_teachers_and_trainers_in_undergraduate_medical_education___guidance_0815.pdf_56440721.pdf (Accessed on 19th September 2023).

Cooper A, Rodman A. AI and Medical Education - A 21st-Century Pandora’s Box. N Engl J Med. 2023;389(5):385–7.

Chanda SS, Banerjee DN. Omission and commission errors underlying AI failures. AI Soc. 2022;17:1–24.

Narayanan A, Kapoor S. ‘GPT-4 and Professional Benchmarks: The Wrong Answer to the Wrong Question’. Substack newsletter. AI Snake Oil (blog). https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks (Accessed on 19th September 2023).

Brainard J. November. As scientists face a flood of papers, AI developers aim to help. Science, 21 2023. doi.10.1126/science.adn0669.

Ouyang F, Jiao P. Artificial intelligence in education: the three paradigms. Computers Education: Artif Intell. 2021;2:100020.

Gibson D, Kovanovic V, Ifenthaler D, Dexter S, Feng S. Learning theories for artificial intelligence promoting learning processes. Br J Edu Technol. 2023;54(5):1125–46.

Guerrero DT, Asaad M, Rajesh A, Hassan A, Butler CE. Advancing Surgical Education: the Use of Artificial Intelligence in Surgical Training. Am Surg. 2023;89(1):49–54.

Lee S. AI tools for educators. EIT InnoEnergy Master School Teachers Conference. 2023. https://www.slideshare.net/ignatia/ai-toolkit-for-educators?from_action=save (Accessed on 24th September 2023).

Download references

Author information

Authors and affiliations.

Department of Pharmacology & Therapeutics, College of Medicine & Medical Sciences, Arabian Gulf University, Manama, Kingdom of Bahrain

Kannan Sridharan & Reginald P. Sequeira

You can also search for this author in PubMed   Google Scholar

Contributions

RPS– Conceived the idea; KS– Data collection and curation; RPS and KS– Data analysis; RPS and KS– wrote the first draft and were involved in all the revisions.

Corresponding author

Correspondence to Kannan Sridharan .

Ethics declarations

Ethics approval and consent to participate.

Not applicable as neither there was any interaction with humans, nor any personal data was collected in this research study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Sridharan, K., Sequeira, R.P. Artificial intelligence and medical education: application in classroom instruction and student assessment using a pharmacology & therapeutics case study. BMC Med Educ 24 , 431 (2024). https://doi.org/10.1186/s12909-024-05365-7

Download citation

Received : 26 September 2023

Accepted : 28 March 2024

Published : 22 April 2024

DOI : https://doi.org/10.1186/s12909-024-05365-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical education
• Pharmacology
• Therapeutics

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 01 May 2024

A critical assessment of using ChatGPT for extracting structured data from clinical notes

• Jingwei Huang   ORCID: orcid.org/0000-0003-2155-6107 1 ,
• Donghan M. Yang 1 ,
• Ruichen Rong 1 ,
• Kuroush Nezafati   ORCID: orcid.org/0000-0002-6785-7362 1 ,
• Colin Treager 1 ,
• Zhikai Chi   ORCID: orcid.org/0000-0002-3601-3351 2 ,
• Shidan Wang   ORCID: orcid.org/0000-0002-0001-3261 1 ,
• Xian Cheng 1 ,
• Yujia Guo 1 ,
• Laura J. Klesse 3 ,
• Guanghua Xiao 1 ,
• Eric D. Peterson 4 ,
• Xiaowei Zhan 1 &
• Yang Xie   ORCID: orcid.org/0000-0001-9456-1762 1  

npj Digital Medicine volume  7 , Article number:  106 ( 2024 ) Cite this article

38 Altmetric

Metrics details

• Non-small-cell lung cancer

Existing natural language processing (NLP) methods to convert free-text clinical notes into structured data often require problem-specific annotations and model training. This study aims to evaluate ChatGPT’s capacity to extract information from free-text medical notes efficiently and comprehensively. We developed a large language model (LLM)-based workflow, utilizing systems engineering methodology and spiral “prompt engineering” process, leveraging OpenAI’s API for batch querying ChatGPT. We evaluated the effectiveness of this method using a dataset of more than 1000 lung cancer pathology reports and a dataset of 191 pediatric osteosarcoma pathology reports, comparing the ChatGPT-3.5 (gpt-3.5-turbo-16k) outputs with expert-curated structured data. ChatGPT-3.5 demonstrated the ability to extract pathological classifications with an overall accuracy of 89%, in lung cancer dataset, outperforming the performance of two traditional NLP methods. The performance is influenced by the design of the instructive prompt. Our case analysis shows that most misclassifications were due to the lack of highly specialized pathology terminology, and erroneous interpretation of TNM staging rules. Reproducibility shows the relatively stable performance of ChatGPT-3.5 over time. In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively. Our study shows the feasibility of using ChatGPT to process large volumes of clinical notes for structured information extraction without requiring extensive task-specific human annotation and model training. The results underscore the potential role of LLMs in transforming unstructured healthcare data into structured formats, thereby supporting research and aiding clinical decision-making.

Similar content being viewed by others

Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks

Assessing ChatGPT 4.0’s test performance and clinical diagnostic accuracy on USMLE STEP 2 CK and clinical case reports

Large language models streamline automated machine learning for clinical studies

Introduction.

Large Language Models (LLMs) 1 , 2 , 3 , 4 , 5 , 6 , such as Generative Pre-trained Transformer (GPT) models represented by ChatGPT, are being utilized for diverse applications across various sectors. In the healthcare industry, early applications of LLMs are being used to facilitate patient-clinician communication 7 , 8 . To date, few studies have examined the potential of LLMs in reading and interpreting clinical notes, turning unstructured texts into structured, analyzable data.

Traditionally, the automated extraction of structured data elements from medical notes has relied on medical natural language processing (NLP) using rule-based or machine-learning approaches or a combination of both 9 , 10 . Machine learning methods 11 , 12 , 13 , 14 , particularly deep learning, typically employ neural networks and the first generation of transformer-based large language models (e.g., BERT). Medical domain knowledge needs to be integrated into model designs to enhance performance. However, a significant obstacle to developing these traditional medical NLP algorithms is the limited existence of human-annotated datasets and the costs associated with new human annotation 15 . Despite meticulous ground-truth labeling, the relatively small corpus sizes often result in models with poor generalizability or make evaluations of generalizability impossible. For decades, conventional artificial intelligence (AI) systems (symbolic and neural networks) have suffered from a lack of general knowledge and commonsense reasoning. LLMs, like GPT, offer a promising alternative, potentially using commonsense reasoning and broad general knowledge to facilitate language processing.

ChatGPT is the application interface of the GPT model family. This study explores an approach to using ChatGPT to extract structured data elements from unstructured clinical notes. In this study, we selected lung cancer pathology reports as the corpus for extracting detailed diagnosis information for lung cancer. To accomplish this, we developed and improved a prompt engineering process. We then evaluated the effectiveness of this method by comparing the ChatGPT output with expert-curated structured data and used case studies to provide insights into how ChatGPT read and interpreted notes and why it made mistakes in some cases.

Data and endpoints

The primary objective of this study was to develop an algorithm and assess the capabilities of ChatGPT in processing and interpreting a large volume of free-text clinical notes. To evaluate this, we utilized unstructured lung cancer pathology notes, which provide diagnostic information essential for developing treatment plans and play vital roles in clinical and translational research. We accessed a total of 1026 lung cancer pathology reports from two web portals: the Cancer Digital Slide Archive (CDSA data) ( https://cancer.digitalslidearchive.org/ ) and The Cancer Genome Atlas (TCGA data) ( https://cBioPortal.org ). These platforms serve as public data repositories for de-identified patient information, facilitating cancer research. The CDSA dataset was utilized as the “training” data for prompt development, while the TCGA dataset, after removing the overlapping cases with CDSA, served as the test data for evaluating the ChatGPT model performance.

From all the downloaded 99 pathology reports from CDSA for the training data, we excluded 21 invalid reports due to near-empty content, poor scanning quality, or missing report forms. Seventy-eight valid pathology reports were included as the training data to optimize the prompt. To evaluate the model performance, 1024 pathology reports were downloaded from cBioPortal. Among them, 97 overlapped with the training data and were excluded from the evaluation. We further excluded 153 invalid reports due to near-empty content, poor scanning quality, or missing report forms. The invalid reports were preserved to evaluate ChatGPT’s handling of irregular inputs separately, and were not included in the testing data for accuracy performance assessment. As a result, 774 valid pathology reports were included as the testing data for performance evaluation. These valid reports still contain typos, missing words, random characters, incomplete contents, and other quality issues challenging human reading. The corresponding numbers of reports used at each step of the process are detailed in Fig. 1 .

Exclusions are accounted for due to reasons such as empty reports, poor scanning quality, and other factors, including reports of stage IV or unknown conditions.

The specific task of this study was to identify tumor staging and histology types which are important for clinical care and research from pathology reports. The TNM staging system 16 , outlining the primary tumor features (T), regional lymph node involvement (N), and distant metastases (M), is commonly used to define the disease extent, assign prognosis, and guide lung cancer treatment. The American Joint Committee on Cancer (AJCC) has periodically released various editions 16 of TNM classification/staging for lung cancers based on recommendations from extensive database analyses. Following the AJCC guideline, individual pathologic T, N, and M stage components can be summarized into an overall pathologic staging score of Stage I, II, III, or IV. For this project, we instructed ChatGPT to use the AJCC 7 th edition Cancer Staging Manual 17 as the reference for staging lung cancer cases. As the lung cancer cases in our dataset are predominantly non-metastatic, the pathologic metastasis (pM) stage was not extracted. The data elements we chose to extract and evaluate for this study are pathologic primary tumor (pT) and pathologic lymph node (pN) stage components, overall pathologic tumor stage, and histology type.

Overall Performance

Using the training data in the CDSA dataset ( n  = 78), we experimented and improved prompts iteratively, and the final prompt is presented in Fig. 2 . The overall performance of the ChatGPT (gpt-3.5-turbo-16k model) is evaluated in the TCGA dataset ( n  = 774), and the results are summarized in Table 1 . The accuracy of primary tumor features (pT), regional lymph node involvement (pN), overall tumor stage, and histological diagnosis are 0.87, 0.91, 0.76, and 0.99, respectively. The average accuracy of all attributes is 0.89. The coverage rates for pT, pN, overall stage and histological diagnosis are 0.97, 0.94, 0.94 and 0.96, respectively. Further details of the accuracy evaluation, F1, Kappa, recall, and precision for each attribute are summarized as confusion matrices in Fig. 3 .

Final prompt for information extraction and estimation from pathology reports.

For meaningful evaluation, the cases with uncertain values, such as “Not Available”, “Not Specified”, “Cannot be determined”, “Unknown”, et al. in reference and prediction have been removed. a Primary tumor features (pT), b regional lymph node involvement (pN), c overall tumor stage, and d histological diagnosis.

Inference and Interpretation

To understand how ChatGPT reads and makes inferences from pathology reports, we demonstrated a case study using a typical pathology report in this cohort (TCGA-98-A53A) in Fig. 4a . The left panel shows part of the original pathology report, and the right panel shows the ChatGPT output with estimated pT, pN, overall stage, and histology diagnosis. For each estimate, ChatGPT gives the confidence level and the corresponding evidence it used for the estimation. In this case, ChatGPT correctly extracted information related to tumor size, tumor features, lymph node involvement, and histology information and used the AJCC staging guidelines to estimate tumor stage correctly. In addition, the confidence level, evidence interpretation, and case summary align well with the report and pathologists’ evaluations. For example, the evidence for the pT category was described as “The pathology report states that the tumor is > 3 cm and < 5 cm in greatest dimension, surrounded by lung or visceral pleura.” The evidence for tumor stage was described as “Based on the estimated pT category (T2a) and pN category (N0), the tumor stage is determined to be Stage IB according to AJCC7 criteria.” It shows that ChatGPT extracted relevant information from the note and correctly inferred the pT category based on the AJCC guideline (Supplementary Fig. 1 ) and the extracted information.

a TCGA-98-A53A. An example of a scanned pathological report (left panel) and ChatGPT output and interpretation (right panel). All estimations and support evidence are consistent with the pathologist’s evaluations. b The GPT model correctly inferred pT as T2a based on the tumor’s size and involvement according to AJCC guidelines.

In another more complex case, TCGA-50-6590 (Fig. 4b ), ChatGPT correctly inferred pT as T2a based on both the tumor’s size and location according to AJCC guidelines. Case TCGA-44-2656 demonstrates a more challenging scenario (Supplementary Fig. 2 ), where the report only contains some factual data without specifying pT, pN, and tumor stage. However, ChatGPT was able to infer the correct classifications based on the reported facts and provide proper supporting evidence.

Error analysis

To understand the types and potential reasons for misclassifications, we performed a detailed error analysis by looking into individual attributes and cases where ChatGPT made mistakes, the results of which are summarized below.

Primary tumor feature (pT) classification

In total, 768 cases with valid reports and reference values in the testing data were used to evaluate the classification performance of pT. Among them, 15 cases were reported with unknown or empty output by ChatGPT, making the coverage rate 0.97. For the remaining 753 cases, 12.6% of pT was misclassified. Among these misclassification cases, the majority were T1 misclassified as T2 (67 out of 753 or 8.9%) or T3 misclassified as T2 (12 out of 753, or 1.6%).

In most cases, ChatGPT extracted the correct tumor size information but used an incorrect rule to distinguish pT categories. For example, in the case TCGA-22-4609 (Fig. 5a ), ChatGPT stated, “Based on the tumor size of 2.0 cm, it falls within the range of T2 category according to AJCC 7th edition for lung carcinoma staging manual.” However, according to the AJCC 7 th edition staging guidelines for lung cancer, if the tumor is more than 2 cm but less than 3 cm in greatest dimension and does not invade nearby structures, pT should be classified as T1b. Therefore, ChatGPT correctly extracted the maximum tumor dimension of 2 cm but incorrectly interpreted this as meeting the criteria for classification as T2. Similarly, for case TCGA-85-A4JB, ChatGPT incorrectly claimed, “Based on the tumor size of 10 cm, the estimated pT category is T2 according to AJCC 7th edition for lung carcinoma staging manual.” According to the AJCC 7 th edition staging guidelines, a tumor more than 7 cm in greatest dimension should be classified as T3.

a TCGA-22-4609 illustrates a typical case where the GPT model uses a false rule, which is incorrect by AJCC guideline. b Case TCGA-39-5028 shows a complex case where there exist two tumors and the GPT model only capture one of them. c Case TCGA-39-5016 reveals a case where the GPT model made a mistake for getting confused with domain terminology.

Another challenging situation arose when multiple tumor nodules were identified within the lung. In the case of TCGA-39-5028 (Fig. 5b ), two separate tumor nodules were identified: one in the right upper lobe measuring 2.1 cm in greatest dimension and one in the right lower lobe measuring 6.6 cm in greatest dimension. According to the AJCC 7 th edition guidelines, the presence of separate tumor nodules in a different ipsilateral lobe results in a classification of T4. However, ChatGPT classified this case as T2a, stating, “The pathology report states the tumor’s greatest diameter as 2.1 cm”. This classification would be appropriated if the right upper lobe nodule were a single isolated tumor. However, ChatGPT failed to consider the presence of the second, larger nodule in the right lower lobe when determining the pT classification.

Regional lymph node involvement (pN)

The classification performance of pN was evaluated using 753 cases with valid reports and reference values in the testing data. Among them, 27 cases were reported with unknown or empty output by ChatGPT, making the coverage rate 0.94. For the remaining 726 cases, 8.5% of pN was misclassified. Most of these misclassification cases were N1 misclassified as N2 (32 cases). The AJCC 7th edition staging guidelines use the anatomic locations of positive lymph nodes to determine N1 vs. N2. However, most of the misclassification cases were caused by ChatGPT interpreting the number of positive nodes rather than the locations of the positive nodes. One such example is the case TCGA-85-6798. The report states, “Lymph nodes: 2/16 positive for metastasis (Hilar 2/16)”. Positive hilar lymph nodes correspond to N1 classification according to AJCC 7th edition guidelines. However, ChatGPT misclassifies this case as N2, stating, “The pathology report states that 2 out of 16 lymph nodes are positive for metastasis. Based on this information, the pN category can be estimated as N2 according to AJCC 7th edition for lung carcinoma staging manual.” This interpretation is incorrect, as the number of positive lymph nodes is not part of the criteria used to determine pN status according to AJCC 7th edition guidelines. The model misinterpreted pN2 predictions in 22 cases due to similar false assertions.

In some cases, the ChatGPT model made classification mistakes by misunderstanding the locations’ terminology. Figure 5c shows a case (TCGA-39-5016) where the ChatGPT model recognized that “6/9 peribronchial lymph nodes involved, “ corresponding with classification as N1, but ChatGPT misclassified this case as N2. By AJCC 7th edition guidelines, N2 is defined as “Metastasis in ipsilateral mediastinal and/or subcarinal lymph node(s)”. The ChatGPT model did not fully understand that terminology and made misclassifications.

Pathology tumor stage

The overall tumor stage classification performance was evaluated using 744 cases with valid reports and reference values as stage I, II and III in the testing data. Among them, 18 cases were reported as unknown or empty output by ChatGPT making the coverage rate as 0.94. For the remaining 726 cases, 23.6% of the overall stage was misclassified. Since the overall stage depends on individual pT and pN stages, the mistakes could come from misclassification of pT or pN (error propagation) or applying incorrect inference rules to determine the overall stage from pT and pN (incorrect rules). Looking into the 56 cases where ChatGPT misclassified stage II as stage III, 22 cases were due to error propagation, and 34 were due to incorrect rules. Figure 6a shows an example of error propagation (TCGA-MP-A4TK). ChatGPT misclassified the pT stage from T2a to T3, and then this mistake led to the incorrect classification of stage IIA to stage IIIA. Figure 6b illustrates a case (TCGA-49-4505) where ChatGPT made correct estimation of pT and pN but made false prediction about tumor stage by using a false rule. Among the 34 cases affected by incorrect rules, ChatGPT mistakenly inferred tumor stage as stage III for 26 cases where pT is T3 and pN is N0, respectively. For example, for case TCGA-55-7994, ChatGPT provided the evidence as “Based on the estimated pT category (T3) and pN category (N0), the tumor stage is determined to be Stage IIIA according to AJCC7 criteria”. According to AJCC7, tumors with T3 and N0 should be classified as stage IIB. Similarly, error analysis for other tumor stages shows that misclassifications come from both error propagation and applying false rules.

a Case TCGA-MP-A4TK: An example of typical errors GPT made in the experiments, i.e. GPT took false rule and further led to faulty propagation. b Case TCGA-49-4505: The GPT model made false estimation of Stage IIIA with a false rule, although it made correct inference with T2b and N1.

Histological diagnosis

The classification performance of histology diagnosis was evaluated using 762 cases with valid reports and reference values in the testing data. Among them, 17 cases were reported as either unknown or empty output by ChatGPT, making the coverage rate 0.96. For the remaining 745 cases, 6 ( < 1%) of histology types were misclassified. Among the mistakes that ChatGPT made for histology diagnosis, ChatGPT misclassified 3 of them as “other” type and 3 cases of actual “other” type (neither adenocarcinomas nor squamous cell carcinomas) as 2 adenocarcinomas and 1 squamous cell carcinoma. In TCGA-22-5485, two tumors exist: one squamous cell carcinoma and another adenocarcinoma, which should be classified as the ‘other’ type. However, ChatGPT only identified and extracted information for one tumor. In the case TCGA-33-AASB, which is the “other” type of histology, ChatGPT captured the key information and gave it as evidence: “The pathology report states the histologic diagnosis as infiltrating poorly differentiated non-small cell carcinoma with both squamous and glandular features”. However, it mistakenly estimated this case as “adenocarcinoma”. In another case (TCGA-86-8668) of adenocarcinoma, ChatGPT again captured key information and stated as evidence, “The pathology report states the histologic diagnosis as Bronchiolo-alveolar carcinoma, mucinous” but could not tell it is a subtype of adenocarcinoma. Both cases reveal that ChatGPT still has limitations in the specific domain knowledge in lung cancer pathology and the capability of correcting understanding its terminology.

Analyzing irregularities

The initial model evaluation and prompt-response review uncovered irregular scenarios: the original pathology reports may be blank, poorly scanned, or simply missing report forms. We reviewed how ChatGPT responded to these anomalies. First, when a report was blank, the prompt contained only the instruction part. ChatGPT failed to recognize this situation in most cases and inappropriately generated a fabricated case. Our experiments showed that, with the temperature set at 0 for blank reports, ChatGPT converged to a consistent, hallucinated response. Second, for nearly blank reports with a few random characters and poorly scanned reports, ChatGPT consistently converged to the same response with increased variance as noise increased. In some cases, ChatGPT responded appropriately to all required attributes but with unknown values for missing information. Last, among the 15 missing report forms in a small dataset, ChatGPT responded “unknown” as expected in only 5 cases, with the remaining 10 still converging to the hallucinated response.

Reproducibility evaluation

Since ChatGPT models (even with the same version) evolve over time, it is important to evaluate the stability and reproducibility of ChatGPT. For this purpose, we conducted experiments with the same model (“gpt-3.5-turbo-0301”), the same data, prompt, and settings (e.g., temperature = 0) twice in early April and the middle of May of 2023. The rate of equivalence between ChatGPT estimations in April and May on key attributes of interest (pT, pN, tumor stage, and histological diagnosis) is 0.913. The mean absolute error between certainty degrees in the two experiments is 0.051. Considering the evolutionary nature of ChatGPT models, we regard an output difference to a certain extent as reasonable and the overall ChatGPT 3.5 model as stable.

Comparison with other NLP methods

In order to have a clear perspective on how ChatGPT’s performance stands relative to established methods, we conducted a comparative analysis of the results generated by ChatGPT with two established methods: a keyword search algorithm and a deep learning-based Named Entity Recognition (NER) method.

Data selection and annotation

Since the keyword search and NER methods do not support zero-shot learning and require human annotations on the entity level, we carefully annotated our dataset for these traditional NLP methods. We used the same training and testing datasets as in the prompt engineering for ChatGPT. The training dataset underwent meticulous annotation by experienced medical professionals, adhering to the AJCC7 standards. This annotation process involved identifying and highlighting all relevant entities and text spans related to stage, histology, pN, and pT attributes. The detailed annotation process for the 78 cases required a few weeks of full-time work from medical professionals.

Keyword search algorithm using wordpiece tokenizer

For the keyword search algorithm, we employed the WordPiece tokenizer to segment words into subwords. We compiled an annotated entity dictionary from the training dataset. To assess the performance of this method, we calculated span similarities between the extracted spans in the validation and testing datasets and the entries in the dictionary.

Named Entity Recognition (NER) classification algorithm

For the NER classification algorithm, we designed a multi-label span classification model. This model utilized the pre-trained Bio_ClinicalBERT as its backbone. To adapt it for multi-label classification, we introduced an additional linear layer. The model underwent fine-tuning for 1000 epochs using the stochastic gradient descent (SGD) optimizer. The model exhibiting the highest overall F1 score on the validation dataset was selected as the final model for further evaluation in the testing dataset.

Performance evaluation

We evaluated the performance of both the keyword search and NER methods on the testing dataset. We summarized the predicted entities/spans and their corresponding labels. In cases where multiple related entities were identified for a specific category, we selected the most severe entities as the final prediction. Moreover, we inferred the stage information for corpora lacking explicit staging information by aggregating details from pN, pT, and diagnosis, aligning with the AJCC7 protocol. The overall predictions for stage, diagnosis, pN, and pT were compared against the ground truth table to gauge the accuracy and effectiveness of our methods. The results (Supplementary Table S1 ) show that the ChatGPT outperforms WordPiece tokenizer and NER Classifier. The average accuracy for ChatGPT, WordPiece tokenizer, and NER Classifier are 0.89, 0.51, and 0.76, respectively.

Prompt engineering process and results

Prompt design is a heuristic search process with many elements to consider, thus having a significantly large design space. We conducted many experiments to explore better prompts. Here, we share a few typical prompts and the performance of these prompts in the training data set to demonstrate our prompt engineering process.

Output format

The most straightforward prompt without special design would be: “read the pathology report and answer what are pT, pN, tumor stage, and histological diagnosis”. However, this simple prompt would make ChatGPT produce unstructured answers varying in format, terminology, and granularity across the large number of pathology reports. For example, ChatGPT may output pT as “T2” or “pT2NOMx”, and it outputs histological diagnosis as “Multifocal invasive moderately differentiated non-keratinizing squamous cell carcinoma”. The free-text answers will require a significant human workload to clean and process the output from ChatGPT. To solve this problem, we used a multiple choice answer format to force ChatGPT to pick standardized values for some attributes. For example, for pT, ChatGPT could only provide the following outputs: “T0, Tis, T1, T1a, T1b, T2, T2a, T2b, T3, T4, TX, Unknown”. For the histologic diagnosis, ChatGPT could provide output in one of these categories: Lung Adenocarcinoma, Lung Squamous Cell Carcinoma, Other, Unknown. In addition, we added the instruction, “Please make sure to output the whole set of answers together as a single JSON file, and don’t output anything beyond the required JSON file,” to emphasize the requirement for the output format. These requests in the prompt make the downstream analysis of ChatGPT output much more efficient. In order to know the certainty degree of ChatGPT’s estimate and the evidence, we asked ChatGPT to provide the following 4 outputs for each attribute/variable: extracted value as stated in the pathology report, estimated value based on AJCC 7th edition for lung carcinoma staging manual, the certainty degree of the estimation, and the supporting evidence for the estimation. The classification accuracy of this prompt with multiple choice output format (prompt v1) in our training data could achieve 0.854.

Evidence-based inference

One of the major concerns for LLM is that the results from the model are not supported by any evidence, especially when there is not enough information for specific questions. In order to reduce this problem, we emphasize the use of evidence for inference in the prompt by adding this instruction to ChatGPT: “Please ensure to make valid inferences for attribute estimation based on evidence. If there is no available evidence provided to make an estimation, please answer the value as “Unknown.” In addition, we asked ChatGPT to “Include “comment” as the last key of the JSON file.” After adding these two instructions (prompt v2), the performance of the classification in the training data increased to 0.865.

Chain of thought prompting by asking intermediate questions

Although tumor size is not a primary interest for diagnosis and clinical research, it plays a critical role in classifying the pT stage. We hypothesize that if ChatGPT pays closer attention to tumor size, it will have better classification performance. Therefore, we added an instruction in the prompt (prompt v3) to ask ChatGPT to estimate: “tumor size max_dimension: [<the greatest dimension of tumor in Centimeters (cm)>, ‘Unknown’]” as one of the attributes. After this modification, the performance of the classification in the training data increased to 0.90.

Providing examples

Providing examples is an effective way for humans to learn, and it should have similar effects for ChatGPT. We provided a specific example to infer the overall stage based on pT and pN by adding this instruction: “Please estimate the tumor stage category based on your estimated pT category and pN category and use AJCC7 criteria. For example, if pT is estimated as T2a and pN as N0, without information showing distant metastasis, then by AJCC7 criteria, the tumor stage is “Stage IB”.” After this modification (prompt v4), the performance of the classification in the training data increased to 0.936.

Although we can further refine and improve prompts, we decided to use prompt v4 as the final model and apply it to the testing data and get the final classification accuracy of 0.89 in the testing data.

ChatGPT-4 performance

LLM evolves rapidly and OpenAI just released the newest GPT-4 Turbo model (GPT-4-1106-preview) in November 2023. To compare this new model with GPT-3.5-Turbo, we applied this newest GPT model GPT-4-1106 to analyze all the lung cancer pathology notes in the testing data. The classification result and the comparison with the GPT-3.5-Turbo-16k are summarized in Supplementary Table 1 . The results show that GPT-4-turbo performs better in almost every aspect; overall, the GPT-4-turbo model increases performance by over 5%. However, GPT-4-Turbo is much more expensive than GPT-3.5-Turbo. The performance of GPT-3.5-Turbo-16k is still comparable and acceptable. As such, this study mainly focuses on assessing GPT-3.5-Turbo-16k, but highlights the fast development and promise of using LLM to extract structured data from clinical notes.

Analyzing osteosarcoma data

To demonstrate the broader application of this method beyond lung cancer, we collected and analyzed clinical notes from pediatric osteosarcoma patients. Osteosarcoma, the most common type of bone cancer in children and adolescents, has seen no substantial improvement in patient outcomes for the past few decades 18 . Histology grades and margin status are among the most important prognostic factors for osteosarcoma. We collected pathology reports from 191 osteosarcoma cases (approved by UTSW IRB #STU 012018-061). Out of these, 148 cases had histology grade information, and 81 had margin status information; these cases were used to evaluate the performance of the GPT-3.5-Turbo-16K model and our prompt engineering strategy. Final diagnoses on grade and margin were manually reviewed and curated by human experts, and these diagnoses were used to assess ChatGPT’s performance. All notes were de-identified prior to analysis. We applied the same prompt engineering strategy to extract grade and margin information from these osteosarcoma pathology reports. This analysis was conducted on our institution’s private Azure OpenAI platform, using the GPT-3.5-Turbo-16K model (version 0613), the same model used for lung cancer cases. ChatGPT accurately classified both grades (with a 98.6% accuracy rate) and margin status (100% accuracy), as shown in Supplementary Fig. 3 . In addition, Supplementary Fig. 4 details a specific case, illustrating how ChatGPT identifies grades and margin status from osteosarcoma pathology reports.

Since ChatGPT’s release in November 2022, it has spurred many potential innovative applications in healthcare 19 , 20 , 21 , 22 , 23 . To our knowledge, this is among the first reports of an end-to-end data science workflow for prompt engineering, using, and rigorously evaluating ChatGPT in its capacity of batch-processing information extraction tasks on large-scale clinical report data.

The main obstacle to developing traditional medical NLP algorithms is the limited availability of annotated data and the costs for new human annotations. To overcome these hurdles, particularly in integrating problem-specific information and domain knowledge with LLMs’ task-agnostic general knowledge, Augmented Language Models (ALMs) 24 , which incorporate reasoning and external tools for interaction with the environment, are emerging. Research shows that in-context learning (most influentially, few-shot prompting) can complement LLMs with task-specific knowledge to perform downstream tasks effectively 24 , 25 . In-context learning is an approach of training through instruction or light tutorial with a few examples (so called few-shot prompting; well instruction without any example is called 0-shot prompting) rather than fine-tuning or computing-intensive training, which adjusts model weights. This approach has become a dominant method for using LLMs in real-world problem-solving 24 , 25 , 26 . The advent of ALMs promises to revolutionize almost every aspect of human society, including the medical and healthcare domains, altering how we live, work, and communicate. Our study shows the feasibility of using ChatGPT to extract data from free text without extensive task-specific human annotation and model training.

In medical data extraction, our study has demonstrated the advantages of adopting ChatGPT over traditional methods in terms of cost-effectiveness and efficiency. Traditional approaches often require labor-intensive annotation processes that may take weeks and months from medical professionals, while ChatGPT models can be fine-tuned for data extraction within days, significantly reducing the time investment required for implementation. Moreover, our economic analysis revealed the cost savings associated with using ChatGPT, with processing over 900 pathology reports incurring a minimal monetary cost (less than $10 using GPT 3.5 Turbo and less than $30 using GPT-4 Turbo). This finding underscores the potential benefits of incorporating ChatGPT into medical data extraction workflows, not only for its time efficiency but also for its cost-effectiveness, making it a compelling option for medical institutions and researchers seeking to streamline their data extraction processes without compromising accuracy or quality.

A critical requirement for effectively utilizing an LLM is crafting a high-quality “prompt” to instruct the LLM, which has led to the emergence of an important methodology referred to as “prompt engineering.” Two fundamental principles guide this process: firstly, the provision of appropriate context, and secondly, delivering clear instructions about subtasks and the requirements for the desired response and how it should be presented. For a single query for one-time use, the user can experiment with and revise the prompt within the conversation session until a satisfactory answer is obtained. However, prompt design can become more complex when handling repetitive tasks over many input data files using the OpenAI API. In these instances, a prompt must be designed according to a given data feed while maintaining the generality and coverage for various input data features. In this study, we found that providing clear guidance on the output format, emphasizing evidence-based inference, providing chain of thought prompting by asking for tumor size information, and providing specific examples are critical in improving the efficiency and accuracy of extracting structured data from the free-text pathology reports. The approach employed in this study effectively leverages the OpenAI API for batch queries of ChatGPT services across a large set of tasks with similar input data structures, including but not limited to pathology reports and EHR.

Our evaluation results show that the ChatGPT (gpt-3.5-turbo-16k) achieved an overall average accuracy of 89% in extracting and estimating lung cancer staging information and histology subtypes compared to pathologist-curated data. This performance is very promising because some scanned pathology reports included in this study contained random characters, missing parts, typos, varied formats, and divergent information sections. ChatGPT also outperformed traditional NLP methods. Our case analysis shows that most misclassifications were due to a lack of knowledge of detailed pathology terminology or very specialized information in the current versions of ChatGPT models, which could be avoided with future model training or fine-tuning with more domain-specific knowledge.

While our experiments reveal ChatGPT’s strengths, they also underscore its limitations and potential risks, the most significant being the occasional “hallucination” phenomenon 27 , 28 , where the generated content is not faithful to the provided source content. For example, the responses to blank or near-blank reports reflect this issue, though these instances can be detected and corrected due to convergence towards an “attractor”.

The phenomenon of ‘hallucination’ in LLMs presents a significant challenge in the field. It is important to consider several key factors to effectively address the challenges and risks associated with ChatGPT’s application in medicine. Since the output of an LLM depends on both the model and the prompt, mitigating hallucination can be achieved through improvements in GPT models and prompting strategies. From a model perspective, model architecture, robust training, and fine-tuning on a diverse and comprehensive medical dataset, emphasizing accurate labeling and classification, can reduce misclassifications. Additionally, enhancing LLMs’ comprehension of medical terminology and guidelines by incorporating feedback from healthcare professionals during training and through Reinforcement Learning from Human Feedback (RLHF) can further diminish hallucinations. Regarding prompt engineering strategies, a crucial method is to prompt the GPT model with a ‘chain of thought’ and request an explanation with the evidence used in the reasoning. Further improvements could include explicitly requesting evidence from input data (e.g., the pathology report) and inference rules (e.g., AJCC rules). Prompting GPT models to respond with ‘Unknown’ when information is insufficient for making assertions, providing relevant context in the prompt, or using ‘embedding’ of relevant text to narrow down the semantic subspace can also be effective. Harnessing hallucination is an ongoing challenge in AI research, with various methods being explored 5 , 27 . For example, a recent study proposed “SelfCheckGPT” approach to fact-check black-box models 29 . Developing real-time error detection mechanisms is crucial for enhancing the reliability and trustworthiness of AI models. More research is needed to evaluate the extent, impacts, and potential solutions of using LLMs in clinical research and care.

When considering using ChatGPT and similar LLMs in healthcare, it’s important to thoughtfully consider the privacy implications. The sensitivity of medical data, governed by rigorous regulations like HIPAA, naturally raises concerns when integrating technologies like LLMs. Although it is a less concern to analyze public available de-identified data, like the lung cancer pathology notes used in this study, careful considerations are needed for secured healthcare data. More secured OpenAI services are offered by OpenAI security portal, claimed to be compliant to multiple regulation standards, and Microsoft Azure OpenAI, claimed could be used in a HIPAA-compliant manner. For example, de-identified Osteosarcoma pathology notes were analyzed by Microsoft Azure OpenAI covered by the Business Associate Agreement in this study. In addition, exploring options such as private versions of these APIs, or even developing LLMs within a secure healthcare IT environment, might offer good alternatives. Moreover, implementing strong data anonymization protocols and conducting regular security checks could further protect patient information. As we navigate these advancements, it’s crucial to continuously reassess and adapt appropriate privacy strategies, ensuring that the integration of AI into healthcare is both beneficial and responsible.

Despite these challenges, this study demonstrates our effective methodology in “prompt engineering”. It presents a general framework for using ChatGPT’s API in batch queries to process large volumes of pathology reports for structured information extraction and estimation. The application of ChatGPT in interpreting clinical notes holds substantial promise in transforming how healthcare professionals and patients utilize these crucial documents. By generating concise, accurate, and comprehensible summaries, ChatGPT could significantly enhance the effectiveness and efficiency of extracting structured information from unstructured clinical texts, ultimately leading to more efficient clinical research and improved patient care.

In conclusion, ChatGPT and other LLMs are powerful tools, not just for pathology report processing but also for the broader digital transformation of healthcare documents. These models can catalyze the utilization of the rich historical archives of medical practice, thereby creating robust resources for future research.

Data processing, workflow, and prompt engineering

The lung cancer data we used for this study are publicly accessible via CDSA ( https://cancer.digitalslidearchive.org/ ) and TCGA ( https://cBioPortal.org ), and they are de-identified data. The institutional review board at the University of Texas Southwestern Medical Center has approved this study where patient consent was waived for using retrospective, de-identified electronic health record data.

We aimed to leverage ChatGPT to extract and estimate structured data from these notes. Figure 7a displays our process. First, scanned pathology reports in PDF format were downloaded from TCGA and CDSA databases. Second, R package pdftools, an optical character recognition tool, was employed to convert scanned PDF files into text format. After this conversion, we identified reports with near-empty content, poor scanning quality, or missing report forms, and those cases were excluded from the study. Third, the OpenAI API was used to analyze the text data and extract structured data elements based on specific prompts. In addition, we extracted case identifiers and metadata items from the TCGA metadata file, which was used to evaluate the model performance.

a Illustration of the use of OpenAI API for batch queries of ChatGPT service, applied to a substantial volume of clinical notes — pathology reports in our study. b A general framework for integrating ChatGPT into real-world applications.

In this study, we implemented a problem-solving framework rooted in data science workflow and systems engineering principles, as depicted in Fig. 7b . An important step is the spiral approach 30 to ‘prompt engineering’, which involves experimenting with subtasks, different phrasings, contexts, format specifications, and example outputs to improve the quality and relevance of the model’s responses. It was an iterative process to achieve the desired results. For the prompt engineering, we first define the objective: to extract information on TNM staging and histology type as structured attributes from the unstructured pathology reports. Second, we assigned specific tasks to ChatGPT, including estimating the targeted attributes, evaluating certainty levels, identifying key evidence of each attribute estimation, and generating a summary as output. The output was compiled into a JSON file. In this process, clinicians were actively formulating questions and evaluating the results.

Our study used the “gpt-3.5-turbo” model, accessible via the OpenAI API. The model incorporates 175 billion parameters and was trained on various public and authorized documents, demonstrating specific Artificial General Intelligence (AGI) capabilities 5 . Each of our queries sent to ChatGPT service is a “text completion” 31 , which can be implemented as a single round chat completion. All LLMs have limited context windows, constraining the input length of a query. Therefore, lengthy pathology reports combined with the prompt and ChatGPT’s response might exceed this limit. We used OpenAI’s “tiktoken” Python library to estimate the token count to ensure compliance. This constraint has been largely relaxed by the newly released GPT models with much larger context windows. We illustrate the pseudocode for batch ChatGPT queries on a large pathology report set in Supplementary Fig. 5 .

Model evaluation

We evaluated the performance of ChatGPT by comparing its output with expert-curated data elements provided in the TCGA structured data using the testing data set. Some staging records in the TCGA structured data needed to be updated; our physicians curated and updated those records. To mimic a real-world setting, we processed all reports regardless of data quality to collect model responses. For performance evaluation, we only used valid reports providing meaningful text and excluded the reports with near-empty content, poor scanning quality, and missing report forms, which were reported as irregular cases. We assessed the classification accuracy, F1, Kappa, recall, and precision for each attribute of interest, including pT, pN, overall stage, and histology types, and presented results as accuracy and confusion matrices. Missing data were excluded from the accuracy evaluation, and the coverage rate was reported for predicted values as ‘unknown’ or empty output.

Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Data availability

The lung cancer dataset we used for this study is “Pan-Lung Cancer (TCGA, Nat Genet2016)”, ( https://www.cbioportal.org/study/summary?id=nsclc_tcga_broad_2016 ) and the “luad” and “lusc” subsets from CDSA ( https://cancer.digitalslidearchive.org/ ). We have provided a reference regarding how to access the data 32 . We utilized the provided APIs to retrieve clinical information and pathology reports for the LUAD (lung adenocarcinoma) and LUSC (lung squamous cell carcinoma) cohorts. The pediatric data are the EHR data from UTSW clinic services. The data is available from the corresponding author upon reasonable request and IRB approval.

Code availability

All codes used in this paper were developed using APIs from OpenAI. The prompt for the API is available in Fig. 2 . Method-specific code is available from the corresponding author upon request.

Vaswani, A. et al. Attention is all you need. Adv. Neural Info. Processing Syst. 30 , (2017).

Devlin, J. et al. Bert: Pre-training of deep bidirectional transformers for language understanding . arXiv preprint arXiv:1810.04805, 2018.

Radford, A. et al. Improving language understanding by generative pre-training . OpenAI: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf (2018).

Touvron, H. et al. LLaMA: Open and efficient foundation language models . arXiv preprint arXiv:2302.13971 (2023).

OpenAi, GPT-4 Technical Report . arXiv:2303.08774: https://arxiv.org/pdf/2303.08774.pdf (2023).

Anil, R. et al. Palm 2 technical report . arXiv preprint arXiv:2305.10403 (2023).

Turner, B. E. W. Epic, Microsoft bring GPT-4 to EHRs .

Landi, H. Microsoft’s Nuance integrates OpenAI’s GPT-4 into voice-enabled medical scribe software .

Hao, T. et al. Health Natural Language Processing: Methodology Development and Applications. JMIR Med Inf. 9 , e23898 (2021).

Article   Google Scholar  

Pathak, J., Kho, A. N. & Denny, J. C. Electronic health records-driven phenotyping: challenges, recent advances, and perspectives. J. Am. Med. Inform. Assoc. 20 , e206–e211 (2013).

Article   PubMed   PubMed Central   Google Scholar  

Crichton, G. et al. A neural network multi-task learning approach to biomedical named entity recognition. BMC Bioinforma. 18 , 368 (2017).

Wang, J. et al. Document-Level Biomedical Relation Extraction Using Graph Convolutional Network and Multihead Attention: Algorithm Development and Validation. JMIR Med Inf. 8 , e17638 (2020).

Liu, Y. et al. Roberta: A robustly optimized BERT pretraining approach . arXiv preprint arXiv:1907.11692 (2019).

Rasmy, L. et al. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digit. Med. 4 , 86 (2021).

Wu, H. et al. A survey on clinical natural language processing in the United Kingdom from 2007 to 2022. npj Digit. Med. 5 , 186 (2022).

Amin, M. B. et al. AJCC cancer staging manual . 1024: Springer 2017.

Goldstraw, P. et al. The IASLC Lung Cancer Staging Project: Proposals for the Revision of the TNM Stage Groupings in the Forthcoming (Seventh) Edition of the TNM Classification of Malignant Tumours. J. Thorac. Oncol. 2 , 706–714 (2007).

Article   PubMed   Google Scholar  

Yang, D. M. et al. Osteosarcoma Explorer: A Data Commons With Clinical, Genomic, Protein, and Tissue Imaging Data for Osteosarcoma Research. JCO Clin. Cancer Inform. 7 , e2300104 (2023).

The Lancet Digital, H., ChatGPT: friend or foe? Lancet Digital Health . 5 , e102 (2023).

Nature, Will ChatGPT transform healthcare? Nat. Med. 29 , 505–506 (2023).

Patel, S. B. & Lam, K. ChatGPT: the future of discharge summaries? Lancet Digit. Health 5 , e107–e108 (2023).

Article   CAS   PubMed   Google Scholar  

Ali, S. R. et al. Using ChatGPT to write patient clinic letters. Lancet Digit. Health 5 , e179–e181 (2023).

Howard, A., Hope, W. & Gerada, A. ChatGPT and antimicrobial advice: the end of the consulting infection doctor? Lancet Infect. Dis. 23 , 405–406 (2023).

Mialon, G. et al. Augmented language models: a survey . arXiv preprint arXiv:2302.07842 (2023).

Brown, T. et al. Language Models are Few-Shot Learners . Curran Associates, Inc. (2020).

Wei, J. et al. Chain of thought prompting elicits reasoning in large language models . Adv Neural Inf Processing Syst 35 , 24824–24837 (2022).

Ji, Z. et al. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. 55 , 1–38 (2023).

Alkaissi, H. & S. I. McFarlane, Artificial Hallucinations in ChatGPT: Implications in Scientific Writing. Cureus , (2023).

Manakul, P. A. Liusie, & M. J. F. Gales, SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models . 2023.

Boehm, B. W. A spiral model of software development and enhancement. Computer 21 , 61–72 (1988).

OpenAi. OpenAI API Documentation . Available from: https://platform.openai.com/docs/guides/text-generation .

Gao, J. et al. Integrative analysis of complex cancer genomics and clinical profiles using the cBioPortal. Sci. Signal. 6 , 1–19 (2013).

Download references

Acknowledgements

This work was partially supported by the National Institutes of Health [P50CA70907, R35GM136375, R01GM140012, R01GM141519, R01DE030656, U01CA249245, and U01AI169298], and the Cancer Prevention and Research Institute of Texas [RP230330 and RP180805].

Author information

Authors and affiliations.

Quantitative Biomedical Research Center, Peter O’Donnell School of Public Health, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX, USA 75390, USA

Jingwei Huang, Donghan M. Yang, Ruichen Rong, Kuroush Nezafati, Colin Treager, Shidan Wang, Xian Cheng, Yujia Guo, Guanghua Xiao, Xiaowei Zhan & Yang Xie

Department of Pathology, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX, USA 75390, USA

Department of Pediatrics, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX, USA 75390, USA

Laura J. Klesse

Department of Internal Medicine, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX, USA 75390, USA

Eric D. Peterson

You can also search for this author in PubMed   Google Scholar

Contributions

J.H., Y.X., X.Z. and G.X. designed the study. X.Z., K.N., C.T. and J.H. prepared, labeled, and curated lung cancer datasets. D.M.Y., X.C., Y.G., L.J.K. prepared, labeled, and curated osteosarcoma datasets. Z.C. provided critical inputs as pathologists. Y.X., G.X., E.P. provided critical inputs for the study. J.H. implemented experiments with ChatGPT. R.R. and K.N. implemented experiments with N.L.P. J.H., Y.X., G.X. and S.W. conducted data analysis. Y.X., G.X., J.H., X.Z., D.M.Y. and R.R. wrote the manuscript. All co-authors read and commented on the manuscript.

Corresponding authors

Correspondence to Xiaowei Zhan or Yang Xie .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplemental figures and tables, reporting summary, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Huang, J., Yang, D.M., Rong, R. et al. A critical assessment of using ChatGPT for extracting structured data from clinical notes. npj Digit. Med. 7 , 106 (2024). https://doi.org/10.1038/s41746-024-01079-8

Download citation

Received : 24 July 2023

Accepted : 14 March 2024

Published : 01 May 2024

DOI : https://doi.org/10.1038/s41746-024-01079-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.