case study of quality software

Know more about our vision & mission at Zenkins.

Learn more about our process and how we work?

Contact today! Ready to help you with all tech challenges.

Stay updated with latest technologies and tools.

SAAS Product Development

Empowering Your Vision with Expert SAAS Product Development Services

IT Staff Augmentation

Boosting Your IT Capabilities with Staff Augmentation Solutions

Product Engineering

Crafting Innovation Through Product Engineering Services

Platform and Infrastructure

Elevating Your Digital Foundation: Our Platform and Infrastructure Expertise

Digital Transformation

Unlocking Your Digital Potential: Transformation Services for the Modern Age

Data Engineering

Unlocking the Power of Data: Our Expert Data Engineering Services

IT Consulting

Navigating IT Excellence: Our Comprehensive Consulting Services

• Maintenance And Support

Safeguarding Your Success: Exceptional Maintenance and Support Services

Core Expertise

• .NET Development
• ASP.NET MVC Web Dev.
• ASP.NET Core Web Dev.
• ASP.NET Web Form Dev.
• .NET Windows App Dev.
• .NET WPF Application Dev.
• .NET Core API Development
• .NET WCF Services
• .NET CMS Development
• C# Development
• .NET Core Development
• Full-Stack Web App Dev.
• Agile, Scrum & DevOps
• Integration Services
• Mobile App Development
• AI ML Development
• Legacy .NET App Modernization
• E-Commerce Solutions
• Enterprise Application Dev.
• Database Design And Mgmt
• Cloud Integration
• Azure / AWS / GCP
• Migration Services
• Third-Party Integrations
• UI/ UX Design
• Quality Assurance & Testing
• Security And Compliance
• Consulting And Strategy
• Content Management
• Performance Optimization
• IoT Hardware & Software
• .NET Developers
• C# Developers
• VB.NET Developers
• .NET Core Developers
• Windows Desktop Application Developers
• Azure Developers
• SQL Server Developers
• ASP.NET MVC Developers
• Blazor Developers
• Angular Developers
• Full-Stack .NET Developers
• HTML / CSS Developers
• JavaScript Developers
• Xamarin Developers
• Kentico CMS Developers
• Sitecore Developers
• Umbraco Developers
• Orchard CMS Developers
• DNN (DotNetNuke) Developers
• Sitefinity Developers
• NopCommerce Developers
• mojoPortal Developers
• DevOps Engineers
• UI/UX Designers
• Quality Assurance (QA) Testers
• Security Experts

Build Your Team

We help companies transform, scale, and gain a competitive edge with robust, customized, and innovative digital solutions.

• Technologies
• Entity Framework
• Razor Pages
• Razor Views
• Microservices
• IdentityServer
• Visual Studio
• Visual Studio Code
• Azure DevOps
• SQL Server Management Studio (SSMS)
• AWS (Amazon Web Services)
• GCP (Google Cloud Platform)
• Azure Functions
• Distributed Platforms

Our expertise spans all major technologies and platforms, and advances to innovative technology trends.

Can't find what you need?

• Case Studies

Quality Management Software: A Case Study 

• Post author: Maryliya M J
• Post published: January 19, 2024
• Reading time: 11 mins read

Table of Contents

Quality management plays a critical role in ensuring organizations meet and exceed customer expectations while maintaining operational excellence. To achieve this, many businesses are turning to quality management software solutions. This article presents a case study that explores the implementation and impact of quality management software in an organization. It delves into the challenges faced in quality management, the process of selecting and integrating the software, as well as the benefits and lessons learned from its implementation.

Introduction to Quality Management Software

Quality management is an essential aspect of any organization, ensuring that products or services meet or exceed customer expectations. In today’s fast-paced and competitive business landscape, the need for effective quality management has become even more crucial. Enter quality management software, a powerful tool that can streamline and optimize quality management processes.

Understanding Quality Management

Quality management refers to the systematic processes, procedures, and activities an organization implements to achieve and maintain excellence in its products or services. It involves various aspects, such as quality planning, quality control, quality assurance, and continuous improvement. By focusing on quality management, organizations can enhance customer satisfaction, reduce costs, increase efficiency, and foster a culture of excellence.

Importance of Quality Management Software

Quality management software plays a vital role in enabling organizations to effectively manage and control their quality processes. It provides a centralized platform for capturing, analyzing, and monitoring quality-related data and activities. With quality management software, organizations can automate and streamline various tasks, such as document control, non-conformance management, corrective actions, audits, and supplier management. This software helps in standardizing processes, ensuring compliance with regulations and industry standards, and facilitating continuous improvement initiatives. By leveraging quality management software, organizations can enhance their overall quality management efforts and achieve desired business outcomes.

About the Client

Our client, a manufacturing company, encountered challenges in maintaining product quality standards and effectively tracking defects. Recognizing the critical importance of systematic quality control processes, they sought a Quality Management Software ( QMS ) to enhance their quality assurance procedures. The primary goal was to implement a comprehensive QMS that ensures adherence to quality standards, streamlines quality inspections, and enables efficient tracking of defects. 

Project Overview

The project aimed to develop a robust .NET-based QMS to address the client’s challenges. The primary objectives included implementing modules for quality inspections, defect tracking, and corrective action management. 

The Challenges

• Maintaining Quality Standards: Inconsistent adherence to quality standards was affecting the overall product quality. 
• Defect Tracking Difficulties: Difficulty in tracking defects and implementing timely corrective actions. 
• Lack of Systematic Processes: Absence of a systematic quality control process led to operational inefficiencies and increased defects. 

The Solution

Our team of experienced developers and project managers collaborated to design and implement a comprehensive .NET-based Quality Management Software. The solution included modules for quality inspections, defect tracking, and corrective action management to ensure systematic adherence to quality standards. 

Are you struggling to keep up with your software development needs? Are you looking for a team of dedicated developers who can work on your project full-time and deliver high-quality results? So why wait? Contact us today to learn more about our services and to start the process of hiring your own dedicated development team. Let us help you take your project to the next level! Contact Now!

Key features of the qms.

• Quality Inspections: The QMS facilitated systematic quality inspections to ensure product quality standards were met. 
• Defect Tracking: Comprehensive tools for tracking defects and identifying root causes for timely corrective actions. 
• Corrective Action Management: The software provided a structured approach to corrective action management, addressing issues promptly. 
• Adherence to Quality Standards: Ensured consistent adherence to quality standards through standardized processes. 

The Outcome

The QMS was successfully deployed, resulting in significant improvements in product quality and defect tracking. Systematic quality inspections, defect tracking capabilities, and structured corrective action management contributed to a more streamlined and efficient quality control process. 

Our team’s expertise in developing a tailored Quality Management Software using .NET technologies effectively addressed the client’s challenges. The implementation of quality inspections, defect tracking modules, and corrective action management contributed to a more rigorous and effective quality assurance process. 

By leveraging the software’s capabilities across various departments, organizations can achieve greater operational efficiency, improved customer satisfaction, and enhanced overall performance. However, it is crucial to thoroughly assess the integration requirements and ensure proper training and support are provided to users.

In conclusion, the case study highlights the immense value of quality management software in organizations. By addressing the challenges faced in quality management and implementing an effective software solution, businesses can experience improved efficiency, enhanced quality control, and significant cost reductions.

Are you struggling with maintaining quality standards and defect tracking? Contact us today to explore how our expertise in QMS development can transform your quality control processes and enhance product quality. 

Frequently Asked Questions (FAQ)

1. what is quality management software.

Quality management software is a technological solution designed to assist organizations in effectively managing and improving their quality control processes. It typically includes features such as document control, corrective and preventive actions, audit management, risk assessment, and performance tracking, enabling companies to streamline their quality management practices.

2. How can qMS benefit my organization?

Implementing quality management software can bring numerous benefits to an organization. It enhances operational efficiency, facilitates compliance with industry standards and regulations, reduces the risk of product defects, and improves customer satisfaction by ensuring consistent quality. Additionally, it provides real-time visibility into quality metrics, enabling data-driven decision-making and fostering a culture of continuous improvement.

3. Are there specific industries or sectors that can benefit from quality management software?

QMS can benefit organizations across various industries and sectors. It is particularly valuable for industries with stringent quality requirements, such as manufacturing, healthcare, pharmaceuticals, aerospace, and automotive. However, any organization that values quality, consistency, and process optimization can reap the benefits of implementing quality management software.

4. What are some key considerations for successful implementation of quality management software?

Successful implementation of QMS requires careful planning and execution. Some key considerations include clearly defining quality objectives, involving stakeholders throughout the implementation process, choosing a software solution that aligns with the organization’s needs, providing adequate training to employees, and regularly evaluating and refining quality management processes to ensure ongoing improvement.

Get 50% off on your first project with us! Join our community of satisfied customers and experience the power of our software team today. Contact now and get 50% off your first software project/ product. Don’t miss out on this exclusive offer! Your Name * Your Email * Contact Number * Brief Your Requirement * 0 / 500 Claim Your Discount Please do not fill in this field.

You might also like.

Employee Training Management Software: A Case Study

Event Management Software: A Case Study 

Supply Chain Management Software: A Case Study

Book a free consultation.

Tailored Solutions, Expert Advice, and Project Estimates Await.

Expect a Prompt Call from one of our Account Managers.

• Top .NET Talent
• Time Zone Aligned
• Experienced Team

At Zenkins, we understand the importance of having skilled and dedicated .NET professionals on your side. Our team consists of top .NET talent with a proven track record of delivering high-quality solutions. Whether you need custom .NET application development or require expertise in ASP.NET, C#, or .NET Core, our talented developers are ready to bring your vision to life.

We recognize that time zone differences can often be a challenge in global collaborations. Zenkins addresses this issue by ensuring that our development teams are time zone-aligned with your operations. This means you can expect real-time communication, rapid responses, and a seamless workflow, reducing project delays and ensuring effective collaboration, no matter where you are in the world.

Experience matters in the world of software development, and our team has it in abundance. Zenkins’ experienced team brings a wealth of knowledge and expertise to the table. From legacy .NET application modernization to building cutting-edge cloud-native solutions using the .NET framework, our team has successfully handled a wide array of projects. You can trust us to navigate your software development journey with precision and care.

+91 70690 18504

[email protected]

Zenkins is a leading .NET development company in India, with over 10 years of experience in developing high-quality, scalable, and secure .NET applications. The company has a team of experienced and certified .NET developers who are experts in all aspects of .NET development, including ASP.NET, WPF, Windows Forms, WCF, Xamarin, and more.

• Methodologies
• Our Services
• IT Staffing Services
• Software Outsourcing
• Careers @Zenkins
• Current Openings
• Technology Insights
• Careers Insights

326, Naroda Business Point, Vasant Vihar 2, Nava Naroda, Ahmedabad, Gujarat 382330

Opening soon in Pune Baner, Pune, Maharashtra 411045 - India

Opening soon in Bangalore Vasanth Nagar, Bengaluru, Karnataka, 560020 - India

Privacy Overview

Get 50% off your first project with us.

Join our community of satisfied customers and experience the power of our software team today. Contact now and get 50% off your first software project/ product. Don’t miss out on this exclusive offer!

Article & Observations About Diverse Teams Working Better Together

Software Testing Client Project Case Study

Apr 21 • Case Studies

We are often asked what software testing is . The video below shares a solid definition of the term.

But we thought a software testing project case study might be helpful to better understand what software testers do on a typical day. This includes testing software, writing requirement documents for our clients, and creating user guides to ensure compliance for our clients to use for quality assurance and auditing purposes.

Iterators LLC was hired to complete accessibility testing for a few projects for the Library of Congress (LOC). Accessibility testing is required on all government websites, with Section 508 and WCAG 2.2 requirements used. To become a Trusted Tester an employee must complete the DHS Trusted Tester online training course requirements and pass the DHS Section 508 Trusted Tester Certification Exam of Homeland Security in Accessibility, and we are in a unique position to help on this project. We cross-train all our employees so that we can work on several projects at one time or several different aspects of a project to complete the work and reduce the cost to our clients.

Our first project assigned by LOC was testing their new braille feature on BARD Mobile for Android. We were tasked with testing the braille feature with several refreshable braille displays.

During our testing, we used the Orbit Reader 20 , and two different braille displays from Freedom Focus 14 and Freedom Focus 40 . There are plans to use other refreshable displays such as Humanware, but this testing has not occurred yet. We needed to test refreshable braille displays and their use in tandem with Google BrailleBack and Google TalkBack .

This work was to ensure that all hardware worked as expected with the apps we were testing. For this testing, we had to complete functional testing, smoke testing, exploratory testing and had a user panel to ensure we caught all issues that a visually impaired individual might experience while using the app.

Initially, our client was unsure if we would find any bugs and hesitant to have us enter bugs into Bugzilla as they stated the software was “complicated”. Bugzilla is a web-based general-purpose bug tracking system and not unlike other bug tracking systems we use every day such as Jira, TestRails, PractiTest, and Click-Up.

Testing was completed over several agile sprints with many significant software testing bugs found. Our testing had us test against the National Library Service requirements document. Next, we had to create an up-to-date user manual. While the manual had been updated several times, the testing had not been.

For example, when downloading a book or magazine from the Now Reading section of the mobile app, the download would end up at the bottom of the page. For years, the user guide had listed the download being at the top of the page once the document was downloaded.

Our testing team, on several occasions, said this was an error in the documentation and that the download ends up at the bottom of the page. This was corrected in the user document and sent to the development team to fix per the requirement document.

Over the next several months, we reported 30 high-priority bugs with about half fixed at this point. We have encouraged our client to test in an agile fashion because once the development team is finished, it’s harder to get these bugs fixed.

Our bugs were reported and based on the requirement document used to create the software. Lastly, the user guide had to be rewritten to reflect the app's behavior and general updates.

Once the app was tested and created, the user guide was sent to Communication Services to ensure the style used per other requirement documentation. This document had to be approved before being disseminated to the public. For example, how does the library determine what the Most Popular Books are, and over what period.

Once the document was returned from COS, this PDF document had to be remediated . The process of remediation is to make a PDF, create the heading for the document, alt text added to meaningful images, and decorative images either ignored or taken out of the digital document altogether.

Once the remediation process is complete and validated, the document becomes ADA-compliant. We then provide an accessible PDF that can be read with the use of a screen reader and create the HTML output so that the document can be added to the Library of Congress website.

You can find the current user guide we completed here: https://www.loc.gov/nls/braille-audio-reading-materials/bard-access/bard-mobile-android/#creatingfolders3.3

Case studies can be a great learning tool in software testing and project management. By looking at project case study examples, you can see how the project was planned and executed, as well as how certain tasks were managed. This can give a better understanding of what software testing involves on a daily basis. With the right software testing case studies example, software testers can hone their skills, improve project performance, and ultimately deliver better software testing results.

Related Resources:

• Crafting an Effective Test Plan: A Step-by-Step Guide
• Top Test Management Tools
• Mobile Application Functional and Performance Testing

About the Author

Jill Willcox has worked on accessibility issues for most of her professional career. Iterators is an inclusive women-owned small business (WOSB) certified by the Small Business Administration and WBENC. We provide software testing services for websites, mobile apps, enterprise software, and PDF remediation services, rendering PDFs ADA compliant.

Jill Willcox

Clutch names iterators llc as a top certified women-owned business for 2022.

Iterators LLC named Top Certified Women0owned Business Again 2022

Test Strategy vs Test Plan: What’s the Difference?

What is the difference between a test strategy and a test plan? Read this article to b...

May 05 • Reference

Case Studies

Home » Case Studies

Below are synopses of actual Beta Breakers automation engagements.  In each synopsis however, customer and/or product identities have been scrubbed in order to protect Non-disclosure agreements. If you are interested in similar results for your QA needs, complete the form or call us to get started. 

Case Study #1 – Android Application Acceptance Test Suite

The Android Acceptance Test Suite covers all essential regression features and provides timely feedback and reporting to Android Application developers and QA. This suite is easily run within the CI process or in QA by pulling the most recent HockeyApp build on which to test. By providing constant feedback to QA and reporting regression bugs, the suite saves hours of QA testing per run and allows manual testers to focus on tighter feedback cycles for brand new features. Test results are provided via Cucumber Reports with attached debug information such as custom exception messages, screenshots at the time of the error, and a snapshot of the page XML. Reporting features useful metrics in addition to results, such as percentage of features passed, features failed, skipped, and pending implementation. Utilizing Selenium /Appium, we leverage a single codebase against Google Android and Amazon Fire OS mobile operating systems.  No third party vendors were involved.  The Beta Breakers team consisted of one full-time QE resource with occasional augmented and collaborative support from other Beta Breakers QE staff.  The completed automation suite was delivered on time and was very well accepted.

Case Study #2 – API Acceptance Test Suite

The API Acceptance Test Suite covers all of the exposed endpoints in the customer’s backend API. The test cases are written from the perspective of user stories as opposed to merely unit testing the endpoints. In addition to the backend API, the test suite leverages other dependent APIs. In order to easily facilitate the testing requirements of the Mosaic development team, an HTTP Client was designed and developed for testing REST and REST-like APIs. It currently resides in a Nexus repository and is used by other teams. The Test Suite is fully integrated into the API’s Continuous Integration pipeline on Jenkins. In addition to testing new builds at every deployment, the test suite runs twice a day on a timer. The test suite is configured to accept environmental variables as testing arguments allowing multiple Jenkins jobs to leverage the same test suite while testing multiple environments and features. The Jenkins jobs are configured to conditionally notify the development team of test failures based on the urgency around the features. Developers are notified via email and slack notifications. Test results are stored and displayed on the Jenkins job via the Jenkins Cucumber-JVM plugin. In addition to displaying feature coverage and test results, the reports contain step-by-step user flows with detailed drop downs. These step drop downs convert the HTTP requests made by the test suite into curl commands that the reader can copy and paste directly into their terminal. The reports also contain the REST responses (in this case JSON responses) that were returned via each API call. This approach allows for a high level look at Behavior Driven reports while still allowing readers of the reports to zoom to a lower level view of the flows and what may have gone wrong. This approach also allows many hours of manual QA to be replaced by automation, as well as supplemented by automation. The look and feel of the reports are perfect for giving management a high level look at the quality engineering product velocity while also maintaining low level use for developers and manual QA alike. Baked into the API test suite is a series of tests specifically for monitoring the status of various user-facing processes and service health checks. These test results are sent to an InfluxDB instance where they are read and displayed on another customer product, the “Quality Monitoring Dashboard”.  Again, no third party vendors were involved.  The Beta Breakers team consisted of two full-time QE resources with occasional augmented and collaborative support from other Beta Breakers QE staff.  The completed automation suite was delivered on time and was very well accepted.

Case Study #3 – Web Acceptance Test Suite

The Web Acceptance Test Suite covers all user-facing web features available on the customer’s web application. In addition to covering user features, compatibility testing with various browsers is baked in. Currently, the Web Acceptance test suite measures compatibility with Firefox, Chrome, Safari, Edge, and Internet Explorer. Within this compatibility testing, it also tests video playback and video player functionality. The core engine that drives the Web Acceptance Test suite is Selenium Webdriver and Cucumber. Cucumber allows all test plans to be written and expressed via user stories compatible with Behavior Driven Development. Given the many stages of the development process, and numerous environments, the test suite is parameterized to build and test environments, browsers, and features within the specifications of the user running the test suite. The suite itself manifests in many Jenkins jobs that run the specific tasks required for each environment and browser. The core purpose of the test suite is for complete integration within the development CI pipeline. When developers push new builds to their respective environments, this action triggers a test suite run against those specifications. The results are displayed in HTML Cucumber Reports and the development teams are notified via email and slack. The manual QA team also has access to a number of Jenkins jobs in order to supplement their regression testing, saving hours of manual QA time in the process. The team’s Selenium Grid instance allows for multiple jobs to be running simultaneously against different environments and browsers. This architecture means the manual QA team can run their own regression tests without fear of interfering with the development CI process. Embedded in the test reports are screenshots of test failures as well as links to video playback of the tests in question.

Case Study #4 – iOS Acceptance Test Suite

The iOS Mobile Acceptance Test Suite, similar to the Android Acceptance Test Suite, utilizes our in-house Selenium/Appium framework and covers all essential regression features as well as all other automatable feature tests. The test suite built as a Maven project is integrated into the development CI pipeline. The test suite is configured to test against real devices or iOS simulators; however, due to limitations of the applications requirements, key functionality is not possible on simulators. The test suite thereby will trigger test runs on physical devices located on campus. Test results are provided in the form of Cucumber Reports, via the Jenkins Cucumber-JVM plugin. Embedded in the reports are screenshots of key steps during failure as well as other exception messages, stack traces and other key debug information for developers. The reports themselves provide a good high level view of the user stories which are tested. The test suite is also configured to test against environments throughout the entire development process, including development, staging, QA, and production.

Case Study #5 – Product Acceptance Test Suite

This product (herein referred to as Product A) is available on Web and iOS applications. Our team integrated into the development process, attending sprint planning meetings, daily standups and weekly scrums. Our team developed automated test suites using our in-house Selenium wrapper for both Web and iOS applications. Test suites cover all essential regression features as well as all other automatable feature tests. The Product A iOS Mobile Acceptance Test Suite is configured to test against real devices or iOS simulators; however, due to limitations of the application’s requirements, key functionality is not possible on simulators. The test suite thereby will trigger test runs on physical devices located on campus. Both Web and iOS test suites are integrated within the development CI/CD pipeline. Test results are provided in the form of Cucumber Reports, via the Jenkins Cucumber-JVM plugin. Embedded in the reports are screenshots of key steps during failure as well as other exception messages, stack traces and other key debug information for developers. The reports themselves provide a good high level view of the user stories for which are tested. The test suite is also configured to test against environments throughout the entire development process, including development, staging, QA, and production. In addition to the standard automation of the product, a detailed cost analysis and recommendation was performed and given with regards to the Google App Engine configuration of the product.

Case Study #6 – Product A Performance/Cost Test Suite

In order to properly cost test Product A on the Google App Engine, a parameterized performance test suite needed to be made first in order to simulate the proper user behaviors at expected traffic count. The performance test suite leverages JMeter, using the Maven Blazemeter plugin. The JMeter test plan was written to include the developers’ unit test suite, written in JUnit, in order to simulate user interaction with the site. The JMeter test plan also included manual http requests against features that were not tested in the developers’ unit test suite as user stories. The test suite allows thread count to be passed as an argument by the test runner. A custom sandbox GAE instance was created for the front and back end of the application. Front and back end instances were then tested using the performance test suite, and GAE metrics were measured as the tests were run multiple times against various configurations. For the backend instance, the Cron jobs also had to be taken into consideration. The Cron buckets were filled with user activity and in swoop emptied to allow isolated data analysis of the GAE configuration’s behavior. Once each test was run with each instance type, configurations were further narrowed in on by fine tuning other test parameters. Eventually, an expected cost for each configuration was attained, and a recommendation was made based on the best user experience at the lowest cost.

Case Study #7 – API Acceptance Test Suite

A full regression test was developed for Product A, a collection of REST API endpoints for financial services. The test cases were developed by the automation engineer and finalized collaboratively with the client. A list of bugs and flaws in the behavior of the APIs was delivered. The client did not utilize a CI/CD process, and wished to run the test suite with a manual trigger. Thus, a solution was provided to run the suite via the Maven command line, with an integrated Cucumber HTML report to provide a convenient log of the results for each test run. Technical assistance was provided to configure the Maven project to run securely on a VM. A JMeter load testing script was also developed and provided to the client, with a demonstration of how to use the script on their own, as well as recommendations of load limits from a round of in-house testing.

Case Study #8 – New Product Integration Into A Software Ecosystem

Our customer’s project was to replace a crucial but outdated in-house software application with a customized third-party SAAS solution. The whole of the project included a web front-end SAAS, legacy desktop software, REST APIs, Amazon Web Services and PostgreSQL. Test cases were developed from ‘User Stories’ which documented the end-to-end flows of typical product usage. Since the User Stories could branch into different types of usage or into error conditions, multiple test cases were derived from a single User Story.  Generally, a test case would begin by creating assets via front-end Selenium automation through the third-party SAAS. Next, automated back-end validation of the downstream components was performed, testing the handling, storage and distribution of the information in the assets. Integration tests were also created to test individual components using mocked data. Our customer used an in-house Selenium solution for front-end automation, SmartBear ReadyAPI, and NodeJS with strict linting rules for AWS interactions. Test cases were tied together using Atlassian Bamboo and results were pushed to Practitest. Our automation engineer worked closely with the customer’s QA resources, and integrated into their AGILE workflow, participating in daily SCRUMS, Sprint planning and retrospectives. Our engineer was also responsible for providing live or recorded demonstrations of completed tasks, as well as extensive documentation of the testing behavior.

What Can We Test for You?

" * " indicates required fields

© 2024 Beta Breakers | Privacy Policy

Privacy Policy   |   Contact Us

Beta Breakers

Measuring software quality: a case study

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Unlock the power of Next-gen Strategies for Achieving Efficient SAP Systems on 22nd May 2024

• +1 (214) 347-9928
• [email protected]

Case Studies

• Let's Connect
• Book a Call

Home > Case Studies

• API Testing
• Automation Testing
• Cloud Testing
• Compatibility Testing
• Content QA Testing
• Embedded Testing
• Functional Testing
• Integration Testing
• Load Testing
• Mobile Application Testing
• Performance Testing
• Regression Testing
• Sanity testing
• Security Testing
• Software Testing
• System Testing
• Usability Testing
• Vulnerabilities Testing
• Electronics
• Health and Wellness
• Logistics and Supply Chain
• Manufacturing
• Retail and E-commerce

SAP CRM Performance Testing Solution for India’s Largest Natural Gas Distribution Company

Deployed advanced testing strategies to elevate product quality for a leading it management software provider, reduced software errors 75% faster with end-to-end testing for a leading north american real estate technology provider, revolutionizing restaurant operations: vita mojo's digital transformation with impactqa, supported india's second-largest two-wheeler company in eliminating compatibility and performance bottlenecks, successfully automated workflows for kpmg to ensure process optimization, efficiently managed end-to-end regression workflows for clubspark to achieve 100% automation coverage  , improve user experience by achieving 60% better app performance for a leading global beverages retail/online store, expert testing ability to improve website performance & load capability by 70% for new york road runners (nyrr), get started..

Experience a Free 30 Min Consultation with an Expert.

Schedule a Free Consultation

Subscribe to our newsletter

Get the latest industry news, case studies, blogs and updates directly to your inbox

Invalid Captcha!

Strategies to manage quality requirements in agile software development: a multiple case study

• Open access
• Published: 03 March 2021
• Volume 26 , article number  28 , ( 2021 )

Cite this article

You have full access to this open access article

• Pertti Karhapää   ORCID: orcid.org/0000-0003-1233-772X 1 ,
• Woubshet Behutiye   ORCID: orcid.org/0000-0002-9015-9941 1 ,
• Pilar Rodríguez   ORCID: orcid.org/0000-0002-0618-6104 2 ,
• Markku Oivo   ORCID: orcid.org/0000-0002-1698-2323 1 ,
• Dolors Costal   ORCID: orcid.org/0000-0002-7340-0414 3 ,
• Xavier Franch   ORCID: orcid.org/0000-0001-9733-8830 4 ,
• Sanja Aaramaa 5 ,
• Michał Choraś 6 , 7 ,
• Jari Partanen 8 &
• Antonin Abherve 9  

9086 Accesses

15 Citations

3 Altmetric

Explore all metrics

Agile methods can deliver software that fulfills customer needs rapidly and continuously. Quality requirements (QRs) are important in this regard; however, detailed studies on how companies applying agile methods to manage QRs are limited, as are studies on the rationale for choosing specific QR management practices and related challenges. The aim of this study was to address why practitioners manage QRs as they do and what challenges they face. We also analyzed how existing practices mitigate some of the found challenges. Lastly, we connect the contextual elements of the companies with their practices and challenges. We conducted 36 interviews with practitioners from four companies of varying sizes. Since each company operates in different domains, comparing QR management strategies and related challenges in different contexts was possible. We found that the companies apply proactive, reactive, and interactive strategies to manage QRs. Additionally, our study revealed 40 challenges in six categories that companies applying agile methods may face in QR management. We also identified nine contextual elements that affect QR management practice choices and which, importantly, can explain many related challenges. Based on these findings, we constructed a theoretical model about the connection between context, QR management practices, and challenges. Practitioners in similar contexts can learn from the practices identified in this study. Our preliminary theoretical model can help other practitioners identify what challenges they can expect to face in QR management in different developmental contexts as well as which practices to apply to mitigate these challenges.

Similar content being viewed by others

Agile Quality Requirements Management Best Practices Portfolio: A Situational Method Engineering Approach

Quality Requirements Challenges in the Context of Large-Scale Distributed Agile: An Empirical Study

Using Agile Approaches to Drive Software Process Improvement Initiatives

Avoid common mistakes on your manuscript.

1 Introduction

Quality requirements (QRs) are different from functional requirements (FRs) in the sense that they describe non-functional aspects (e.g., maintainability, reliability, performance, security) of a system (Wagner 2013 ; Wiegers and Beatty 2013 ). QRs are important because even if the promised functionality is delivered, the system will not be regarded as good by users if it does not have, for example, sufficient performance or usability. Agile software development (ASD) is aimed at continuously delivering valuable software to customers. The quality aspects of software are integral to fulfilling customer needs and delivering the promised value. Moreover, considering QRs too late in the development process produces bottlenecks and necessitates further work (Schön et al. 2017 ). It has been found that adopting agile methods often leads to neglecting QRs when the focus is mostly on delivering working functionality to the customer as quickly as possible (Cao and Ramesh 2008 ; Ramesh et al. 2010 ; Curcio et al. 2018 ). Together with the fact that an increasing number of companies, regardless of size, are adopting agile methods (Rodriguez et al. 2012 ; VersionOne Inc. 2016 ), it is important to understand how QRs are or should be managed in ASD. Recent secondary studies on requirements engineering (RE) in ASD have highlighted the importance of QRs and have emphasized that managing QRs in the context of ASD is both challenging and understudied (Schön et al. 2017 ; Heck and Zaidman 2016 ; Heikkilä et al. 2015 ; Inayat et al. 2015 ; Magües et al. 2016 ; Alsaqaf et al. 2017 ; Behutiye et al. 2019 ). Some of these studies—for example, Inayat et al. ( 2015 ) and Alsaqaf et al. ( 2017 )—also concluded that there is a lack of empirical evidence on how QRs are managed in ASD.

Alsaqaf et al. ( 2019 ) is the most recent empirical study to have explored in detail the challenges associated with QRs in large-scale ASD. As one of our case companies applies large-scale ASD, we had the opportunity to compare our findings to those of Alsaqaf et al. We were consequently able to determine whether the same challenges exist in any kind of large-scale ASD, or whether other contributing factors are involved. This consideration underscores the importance of accounting for context while studying the management of QRs. However, in our study of the literature, we discovered that many studies of QR management in ASD focus on specific QRs while overlooking a careful consideration of context.

Our research contributes to the body of knowledge of empirical evidence concerning QR management by examining QR management strategies, practices, and associated challenges in four case companies that apply ASD in different contexts. By differences in context, we mean differences in the characteristics of the setting in which software development occurs, such as company size, organizational model, or type of customers. These characteristics are listed as contextual elements in Table 1 . By QR management strategy, we mean a plan containing different practices to handle QRs; by management practices, we mean the activities that are carried out to handle QRs in order to realize the plan (Merriam-Webster 2019 ). The following three research questions were formulated to determine why practitioners manage QRs as they do, what challenges they face in doing so, and how the contextual elements of the companies are connected with these practices and challenges:

RQ1: What strategies and practices are used to manage QRs in ASD?

RQ2: What challenges are associated with the QR management strategies and practices in ASD?

RQ3: How does context affect QR management strategies, practices, and challenges in ASD?

In this paper, we report on the findings from 36 interviews we conducted with practitioners from four ASD companies. These companies have been using agile methods for several years (from 7 to 13 years). The smallest of the companies employs close to 100 employees; two companies employ close to 1000 employees; and the largest company employs over 100,000 employees globally. The focus of the interviews was on the practitioners’ ASD processes, QR management strategies and practices, and the challenges of QR management. Since the companies examined in our study are of varying sizes and are situated in different contexts, we also investigated the effect of context on QR management strategies and related challenges. Via this analysis, we were able to create an initial theoretical model depicting the relationship between context and QR management challenges as well as the practices employed by the companies to mitigate these challenges.

The main contributions of the study are as follows: (1) a detailed analysis of how the different case companies manage QRs; (2) an in-depth understanding of the actual challenges faced by practitioners when managing QRs in ASD, which is essential for focusing research efforts on generating solutions to actual problems deemed critical from a practitioner’s point of view; (3) a fuller understanding of how contextual elements affect the choice of QR management practices as well as what kinds of challenges the different contextual elements might pose; and (4) a theoretical model of the connections between these contextual elements and QR management challenges and practices. We also discuss our findings in light of those detailed in the existing literature and aggregate the findings in order to facilitate future research on QR management and support practitioners in the management of QRs.

The rest of the paper is organized as follows: Section 2 presents related work. Section 3 describes the research method we followed and introduces the case companies. Section 4 presents our findings regarding the research questions: how the companies applying agile methods manage QRs, what challenges they face, and to what extent contextual elements can explain their choice of practices and the challenges they face. Then, at the end of Section 4 , we present our theoretical model. In Section 5 , we discuss our main findings and relate them to the scientific literature. We also outline some implications for practitioners and researchers before discussing potential threats to the validity of our findings. Section 6 concludes the paper, providing recommendations for future research.

2 Related work

QRs have been defined as requirements related to quality concerns that are not covered by FRs (Pohl 2016 ). As such, QRs have often been referred to as non-functional requirements (NFRs). NFRs have a definition very similar to that of QRs; for example, they cover aspects like performance, reliability, and maintainability, which cannot be covered by FRs (Mylopoulos et al. 1992 ). Other terms have also been used in the scientific literature, like extrafunctional requirements (Shaw 1996 ) or different -ilities (Glass 1998 ). Despite such terminological differences, we have used the term QR for all requirements not designated as FRs.

Although several secondary studies (e.g., Schön et al. 2017 ; Heikkilä et al. 2015 ; Inayat et al. 2015 ) aggregated the research on RE in ASD, focus was placed on requirements in general, and as such no studies, except for Alsaqaf et al. ( 2017 ), exist that describe QR management. For instance, even though Heikkilä et al. ( 2015 ) found that NFRs are often ignored or implemented relying on tacit knowledge, they did not provide a detailed explanation for why this occurs. Likewise, although Inayat et al. ( 2015 ) identified some practices—mainly related to testing activities, such as acceptance tests, system quality tests, and “inspective testing” throughout the lifecycle—used to solve challenges posed by NFRs in ASD, they did not provide details of these solutions; moreover, they reported that the usability of these solutions has not yet been explored. Schön et al. ( 2017 ) found and briefly described ongoing work on QR management in ASD in their secondary study, identifying one framework and two methods for managing QRs in ASD. The framework is called the Agile Framework for Integrating Non-Functional Requirements Engineering (AFFINE). This framework introduces a new role into Scrum that is responsible for NFRs (Bourimi et al. 2010 ). The two methods are (1) Non-functional Requirements Modeling for Agile Process (NORMAP), in which a taxonomy is used to classify requirements as functional or non-functional (Farid 2012 ); and (2) SENoR for NFR elicitation through workshops utilizing the quality sub characteristics of ISO 25010, and the importance of NFRs for the development project (Nawrocki et al. 2014 ). These studies described and proposed practices that can help manage NFRs (or QRs); however, they did not focus on a detailed analysis of the QR management strategies or practices that companies are currently using or on the challenges they are presently facing.

The secondary study by Alsaqaf et al. ( 2017 ) focus specifically on QRs in ASD. They reported on the challenges covered in the existing literature. They summarized the reported QR management practices in large-scale ASD as well as general challenges to QR management in ASD. The authors outlined 12 elements of agile processes that result in diminished focus on QRs, such as the inability of users’ stories to document QRs and the dependency on the product owner (PO) as the single point to collect QRs. The study also inspected different solution proposals for QR management in ASD as well as whether the level of agility of these proposals was considered. They found that the agility factor was considered in only two of the 13 identified proposals. Still, the study did not explain whether the associated challenges were relevant to a specific context. In a more recent empirical study, Alsaqaf et al. ( 2019 ) explored the challenges of engineering QRs in ASD. They revealed the challenges practitioners face when engineering QRs, the mechanisms behind these challenges, and the practices applied to mitigate them. Their research questions were very similar to ours; their focus was on large-scale distributed agile projects in the Netherlands. One of our studied companies also applies distributed agile development in large-scale systems. This gave us the opportunity to compare our findings to those of Alsaqaf et al. Additionally, we included small-to-medium size enterprises (SMEs) in our study, which allowed us to compare practices and challenges at different scales of software development. Alsaqaf et al. ( 2019 ) also discussed the possibility of traceability between QR management challenges and context. However, in their study, context was not modeled in detail and was related mostly to the domain of the companies.

The adoption of agile methods in large-scale development, which was the setting in Alsaqaf et al. ( 2019 ) study, can pose its own set of challenges for the management of requirements, and QRs are no exception. For example, Kalenda et al. ( 2018 ) studied this topic both empirically and in the literature. They found, among other challenges, that the distributed nature of scaled ASD places emphasis on constant communication, and that scaling ASD also means scaling RE, which in turn demands some hierarchy in requirements management. Many of the challenges that arise when scaling ASD or when transforming the plan-driven development of large systems into agile development are both interesting and relevant—that said, this topic is mostly outside the scope of our study. What is relevant for our study is that Kalenda et al. ( 2018 ) determined, from the literature as well as from their empirical results, that when scaling ASD, there is the risk of a loss of quality in code and of an accumulation of technical debt shortly after the transition to scaled agile processes, if the processes are not managed properly. Additionally, Kasauli et al. ( 2017 ) also studied requirements engineering challenges in large-scale ASD. They also found that communication and knowledge management are important in such development, and they additionally emphasized the importance of bridging the gap between customers and developers.

In addition to the mentioned studies on QRs in ASD, we found in our own recent systematic mapping study (SMS) on QR management in agile and rapid software development some proposals for QR management (Behutiye et al. 2019 ). Most of these studies, however, had a narrower scope, focusing only on a particular QR. For example, Azham et al. ( 2011 ), Siponen et al. ( 2005 ), and Rafi et al. ( 2015 ) focused on security in ASD. Azham et al. ( 2011 ) stated that Scrum does not address security issues. Meanwhile, Siponen et al. ( 2005 ) stated, in their work on integrating security into ASD, that current agile methods lack features to specifically address security risks. Rafi et al. ( 2015 ), on the other hand, proposed a modified Scrum method to enhance correctness, usability, and security, and they additionally stated that security and usability are overlooked when focusing on functional features. Usability in ASD is a recurring theme in the literature as well. Butt et al. ( 2014 ), Anwar et al. ( 2014 ), and Singh ( 2008 ) are examples of these studies. Butt et al. ( 2014 ) worked toward integrating usability into ASD and found that usability approaches are not followed in agile approaches when focusing on the development of a running system. Anwar et al. ( 2014 ) reported that ensuring usability is challenging for distributed teams in SMEs that are applying agile methods. Singh’s ( 2008 , p. 556) study on agile methods found that in “traditional Scrum processes […] usability is typically included only as an afterthought, if at all.” Usability and security are among the most frequently studied QRs in the scientific literature we reviewed. As the main focus of these publications was on solution proposals, they did not include detailed descriptions of the challenges faced by companies when managing QRs, nor were they all evaluated in an industrial context.

In our SMS, we also identified some solution proposals focusing not on a specific QR but instead on a wider range of QRs. Farid and Mitropoulos ( 2013 ) proposed the Non-functional Requirements Planning for Agile Processes (NORPLAN) to improve the planning and prioritization of NFRs. Cannizzo et al. ( 2008 ) explained that the implementation of continuous integration (CI) can address QRs, particularly robustness and performance. According to the authors, CI can reveal robustness and performance issues that are not visible in acceptance tests. Similarly, Chen ( 2015 ) showed how CI enhances the reliability of releases. Continuous delivery helps to identify errors in deployment earlier through frequent testing. The study by Singh ( 2008 ) is one of the few to have deeply analyzed challenges. However, the focus of the study was on usability only. Another study that provided a more detailed analysis of challenges was conducted by Cajander et al. ( 2013 ), who also focused on usability as well. The studies that we identified either did not offer a deeper analysis of the challenges faced by companies when managing QRs or had a narrow scope, focusing only on a specific QR.

As a summary, although the literature describes some initial solutions to manage QRs in ASD, the reported proposals often lack empirical validation (Schön et al. 2017 ; Inayat et al. 2015 ; Alsaqaf et al. 2017 ), making it unclear to what degree they are useful in practice or which proposal works in what context. In order to understand and address the challenges that practitioners experience, we must have a better understanding of which challenges are specific to what context and what kinds of practices can mitigate these challenges. This research will shed light on the actual strategies used in practice in different companies, as well as related challenges, and will explore whether—and if so, to what extent—context can explain their choice of strategies or practices and the accompanying challenges.

3 Case study design

We conducted a multiple case study (Runeson et al. 2012 ) of four separate holistic cases. Runeson and Höst’s ( 2009 ) guidelines were adopted for designing and conducting the case studies. We focused on specific units of development within each company, which are explained in Section 3.1 . In Section 3.2 , we describe the data collection procedure and present profiles of all interviewees. Section 3.3 explains the data analysis procedure. In 3.4 we explain the guidelines used for constructing the theoretical model.

3.1 Case and subject selection

The companies in our study are all part of the Q-Rapids Footnote 1 project, which is aimed at improving QR management in ASD. This study is also part of the Q-Rapids project. The companies were selected for the project because of their different domains, sizes, and other contextual factors, as the aim of Q-Rapids is to improve the quality of processes and products in any kind of ASD context. The companies have all applied ASD for several years. Based on the checklist and suggestions provided by Petersen and Wohlin ( 2009 ) and Dybå et al. ( 2012 ), Table 1 presents detailed information about the company contexts in our study so that others can understand and make comparisons to our case companies. We divided context into four different facets, and each facet was further divided into context elements. For example, the Product facet includes the product name (due to confidentiality, the actual names of the products and companies are not given), product type, maturity of the product, and customization, which characterizes the product as a general product, a tailored product, or a product for a specific niche market. The Process facet characterizes the software development process applied in the company, when agile development was adopted, whether or not development is distributed, and the number of developers involved in developing the product. The contextual elements relevant for QR management practices and challenges, as we will show later in the findings, are highlighted in Table 1 with an asterisk (*).

3.1.1 Company A

Company A is an SME developing a long-lived software modeling tool for developers and architects to support software and system engineering. The company is also positioned as a strategic partner of the largest clients of the market in the sectors of Finance, Banking, and Insurance, and it has developed expertise in the field of Digital, Big Data, Analytics, Performance, and Operations. Our case study within Company A was conducted with a software development team in charge of developing one of the modeling language components in that tool. The tool, which has been developed for 25 years, is used by software developers following a model-driven development approach and is also used by the company itself. The development process follows the values of the Agile Manifesto (Fowler and Highsmith 2001 ), emphasizing working software over documentation and utilizing face-to-face communication and close collaboration with customers. The development process is iterative; however, the company does not follow any formalized method, like Scrum. Even though the content of a release is planned every six months, the company prefers to not follow any pre-defined sprint cycles. In addition, even though development relies heavily on face-to-face communication, standard daily or weekly meetings are not practiced. Meetings take place when needed. The focus is on being highly responsive to change, staying ahead of the competition, and fulfilling customer needs, which are the drivers when prioritizing requirements. Requirements are managed in terms of features. Features to be developed are elicited by product management following a feature strategy given by the executive managers and are heavily influenced by customer feedback. The features are communicated to and discussed with the development manager, who in turn plans the features for the sprints together with the developers. Unit tests, nightly integration tests, and validation tests are done before release. All faults are reported through automated reports and discussed between developers, the quality assurance team, and the development manager whenever needed.

3.1.2 Company B

Company B is a large enterprise that develops secure communication and connectivity solutions for multiple industry domains. In a bid to achieve efficiency and shorter time-to-market, the company moved to agile and lean software development in 2007. A well-defined Scrum process, “almost directly from the book,” as one interviewee stated, is followed in the development of highly secure, reliable, and robust embedded devices. The company aims to identify the needs of the market and to fulfill those needs with special solutions. The company has all their development processes well-documented, which is a requirement in the security and safety-critical domain. The development of one of those devices, which started in 2015, is the unit of focus in our case study. The company uses a standardized process for managing the development process as follows. After the project is initiated, product management approves the product proposal. Elicitation of high-level epics follows this approval. In this case, these high-level epics are refined to features and user stories together with developers. Everything is documented in internal wiki-pages. The user stories are passed on to sprint planning, where they are prioritized in the sprint backlog. A sprint is normally two weeks long in a six-month release cycle. Implementation is supported by automated build and testing, manual code review practices, continuous integration, and acceptance testing. An intelligent dashboard is used to visualize and monitor the status of the development process in real time. The company also applies the standard Scrum practices of daily stand-up meetings, sprint reviews, and retrospectives. There is, however, one difference from the common Scrum practices: Due to the special customer segment (i.e., public safety, law enforcement, military, etc.), the company has limited access to the final users of the embedded products and thus limited communication with them during the actual development of the product. Trial users, who can be company-internal, may act as users to test usability prior to release. Feedback from real users is taken into consideration after release, in the maintenance phase.

3.1.3 Company C

Company C is a global company with several business lines in which development is distributed to many locations across the globe. Several coordinated agile teams develop large systems with complex dependencies. Thus, we can consider development in Company C as very large-scale ASD. Due to the size of the company, there is a need for some hierarchy in the decision making and coordination of development done by several teams. Thus, we classify the company as an organization with a hierarchical organization model in the context table (Table 1 ). All the different business lines have a global process description to follow; however, their practices are not identical. Some business lines focus more on software products; while in others, hardware is tightly coupled with the software. The number of employees also varies greatly between business lines. Thus, the process is not exactly the same in all business lines. Even though our main focus of study was on one specific business line, we included interviews from other business lines as well to get a richer picture of QR management in the company. In this paper, we do not go into details of any specific business line but generalize the description. In general, in a business line, development is carried out by teams applying agile methods. Scrum is the most preferred method of the small dynamic teams of nine developers that are formed based on required competences. Features to be built are decided upon and prioritized at the management level, where business opportunities are analyzed. Following that, a separate group of people within that business line is responsible for splitting those features into sub-features and for refining the sub-features into requirements and acceptance criteria. Finally, this same group of people specifies the requirements. Features and sub-features usually require so much effort and different areas of expertise that their development is divided not only into different teams in one location but also into teams at different sites. Thus, one team in one location develops only a small part of a component that is part of a sub-feature, which in turn is part of a feature of the system. Eventually, the POs are responsible for refining the tasks for teams. The PO completes the refinement together with some of the developers, depending on the expertise required for the task. Once this has been done, the PO presents an effort estimation to managers for approval before implementation. Development is accomplished in two-week sprints in either one-month or six-month releases, depending on whether the release is an internal release, a maintenance release, or a customer release. The customer release is typically a six-month release. Testing starts as soon as there is something to test; developers run unit tests before committing code to the CI. A multi-level CI starts with the integration of small parts at the lowest level. Any issues found at this level can be communicated directly back to developers for them to address. The results of several teams are integrated on a higher level in the CI and finally in system testing at the highest level. This practice helps to ensure quality by making it possible to address issues found in lower-level testing as fast as possible.

3.1.4 Company D

Company D is the smallest of the case companies in our study. The company develops customized applications and software solutions for other companies and institutions operating in various areas, such as health care, security, warehouse management, or the space sector. Each project lasts between two and 14 months. The project of focus in our study was in the health care domain. The development employs a mixture of Scrum and Kanban. The company works in close collaboration with customers. Several meetings take place in “scoping sessions” before development, in which the requirements are elicited and an acceptance criterion is formed. The client also participates in intermediate reviews of the product if agreed upon in the contract. The PO sets up the project configuration and development team and defines the tasks for developers to complete in one- to two-week sprints. Sprint length depends on the length of the project but usually lasts one week, from Tuesday to Tuesday. The sprint planning, sprint review, and sprint retrospective meetings are held consecutively within one meeting between sprints. Daily stand-up meetings are also practiced during development. A project Kanban board is utilized to manage the flow of work in the software development process.

3.2 Data collection procedure

Data was collected incrementally through semi-structured interviews by the first two authors. The third author participated in one interview to ensure that the interviews flowed well and for the purpose of researcher triangulation. Interviews were conducted in two rounds. The first round consisted of two sets of interviews. The first set, the initial interviews, were conducted with 13 interviewees in 12 interviews (one interview had two interviewees, see Table 2 in Section 3.2.1 ): three interviewees from Company A, four from Company B, four from Company C, and two from Company D. Based on the initial analysis of these interviews, we identified the need to have complementary interviews at Company C to get a more detailed view of their QR management practices. Company C is much larger than the other companies, and some of the interviewees came from different business lines with slightly different practices. We conducted seven additional interviews in Company C. These complementary interviews used the same interview script as the initial 12 interviews; thus, they were part of the first round of interviews. Later, a second round of interviews was conducted to confirm, complement, and further deepen our knowledge of QR management in the companies. Some of the interview questions in the second round were specific for each case company to capture all relevant information that was not captured during the first interview round. In the second round, we conducted 17 interviews: three interviews from Company A, four from Company B, nine from Company C, and one from Company D. The distribution of interviewees across companies is also depicted in Fig.  1 .

Interview rounds in the different companies

Overall, we conducted 36 interviews with 30 different participants. One interview in the first round included two interviewees, P4 and P5, while other interviewees were interviewed in both the first and second rounds (P1, P2, P4, P7, P12, P19). The interview length for these participants in both round 1 and round 2 are indicated in Table 2 . Those participants who participated in the complementary interviews of round 1 are indicated with a star (*).

Each initial interview was divided into three sections. At the start of the interview, each interviewee was given an informed consent form to read and sign before the recording began. The interview then started with warm-up questions, asked regarding the interviewees’ experience in ASD and the company in question, after which the interviewees were asked about their understanding of QRs. The purpose of this question was to understand how the interviewees perceived QRs and to make sure they understood what we meant by QRs. The rest of the interview questions were defined and structured around different activities of RE according to CMMI for development, version 1.3 (CMMI Product Team 2010 ). We acknowledge that there are varying views on whether CMMI and ASD work together; however, we used CMMI as a comprehensive checklist, not as a maturity or capability model. With the help of CMMI, we sought to ensure that we covered all the different aspects of RE. We asked questions related to how QRs are elicited, how they are prioritized, how they are documented and validated, and so on. In addition, questions were asked about whether the interviewees saw any connection between QRs and other areas related to software development, such as planning and risk management. Finally, in the last section, we wrapped up the interview by asking about any challenges the interviewees faced in the management of QRs in their current development practices. We did not ask about specific challenges that have been reported in the scientific literature in order not to bias the interviewees into thinking that they might have experienced those challenges. In the second round of interviews, we first confirmed the findings of the first round of interviews, after which we asked about specific details that we failed to capture in the first round. The interview scripts can be found in Appendix 1 and Appendix 2 .

The interview scripts were devised by two researchers and reviewed by two additional researchers. All interviews were recorded, stored for internal use by the researchers only, and transcribed for analysis. Transcription was performed by a professional transcription service provider. All data extracted from the interviews were anonymized. We conducted and analyzed all interviews in peers in order to minimize researcher bias.

3.2.1 Subjects

All interviewees were practitioners who were working in software development either directly in the development of products or as a PO or manager with knowledge about QRs or the software development processes. If the company had specific roles related to the quality of the product, these roles was of special interest. Interviewees were selected from varying roles on different levels in the companies to obtain a broad range of perspectives on the management of QRs. From each case company we had one person that was a member of the Q-Rapids project, that we call a champion. The champion helped us in identifying interviewees. In Companies B and D, the champion was a manager; in Company A, the champion was a SW architect; and in Company C, the champion was a transformation expert on processes and tools. A pre-questionnaire was sent to each champion prior to the interviews, in which we asked for domain and context information, as well as for information regarding the methods and tools used in their software life cycle management. The pre-questionnaire was designed to gather initial information for the Q-Rapids project and was also utilized for our case study. Those parts of the pre-questionnaire that were utilized for this case study are presented in Appendix 3 . The pre-questionnaires were also employed for triangulation purposes during data analysis.

Table 2 lists interviewee information for each case company. Those champions who also participated as interviewees are indicated with two stars (**). Note that some interviewees have more than one role. Three of the interviewees have N/A (not applicable) in the experience column. P23 is an expert on development processes but has not worked in software development as a developer. Interviewees P15 and P27 stated that they have worked with software development for longer than they have been employees at the company, but they could not say for how long. Since they both started out in the company as young developers, we can safely assume that their years of experience in software development are equal to their years of experience in the company.

3.3 Analysis procedure

The coding of interview transcripts was completed with NVivo qualitative data analysis software (NVivo 2018 ). The challenges and practices were coded in a deductive (Miles and Huberman 1994 ) manner with a start scheme that was devised a priori by two researchers according to the main topics in the interview questions (ways of working, QR understanding, QR elicitation, QR communication, QR documentation, specifying QRs, QR prioritization, QR validation, and related challenges). The concrete practices and challenges emerged during the analysis of the data. Fig.  2 shows a snapshot of coding in NVivo. On the left-hand side, next to the blue bubbles, are the codes of which elicitation is highlighted, and on the right-hand side there are excerpts of interviews coded as relevant for elicitation of QRs. Identifying the connection of the practices and challenges to context was achieved in an inductive way through thematic analysis (Cruzes and Dybå 2011 ). Contextual elements were identified from the interviews and the pre-questionnaire (the contextual elements are presented in Table 1 ).

NVivo used in coding interview transcripts according to themes in interview questions

Figure 3 shows an example of how we coded parts of interviews related to practices and challenges in the elicitation and analysis of QRs. We coded the practices as shown by text with blue underlining in the yellow boxes and matched them to the activities discussed. Fig. 3 also shows the “together with the client” and “limited access to user scenarios” codes with red underlining. As these were both related to the same concept of access or limited access to users, we classified them under the same theme. Further, we identified contextual elements enabling practices or contributing to inhibiting challenges. In the example in the figure, access to users enables the practice of conducting scoping sessions together with the users and using templates and checklists in the elicitation and analysis of QRs. It also shows how limited access to users poses a challenge for elicitation but can later be mitigated by the practice of value stream mapping sessions (VSM, explained later in Section 4.1.2 ). Interviewee IDs and parts of the text in the challenge box are hidden to ensure confidentiality.

Example of connecting challenges and practices with contextual elements

The information from the pre-questionnaire was also used for triangulation by checking that the information collected through interviews was in agreement. We conducted workshops together with the champions and key roles of the companies to validate the results of our analysis. The number of workshops and participants are presented in Table 3 . During the workshops, which lasted approximately 1.5 h, we presented process models of the companies’ software development, including QR management, followed by an open discussion on whether the model was accurate and properly fit the company’s way of working. In most of the case companies, the input from the workshops required only minor changes in detail regarding, for example, QR management tools and details about QR flows, but these changes did not affect the findings presented in this paper in any other way. In the case of Company C, we needed to update the flow of reporting to correctly order the activities and recurrence timescale. When structuring the findings, we constructed a theoretical model of the relationship between contextual elements and the reported challenges and included also the practices that the companies apply that have the potential to mitigate the challenges. This was done to better highlight the reasons for the chosen practices and how the practices, challenges, and contextual elements were connected. How we did this is presented in Section 3.4 . After we had analyzed the contextual impact, the results were shared with the champions of the companies, who reviewed the results. The results were discussed with the champions when necessary and minor changes to details were done when needed. These results can be used in future research to explore whether the findings are generalizable to other companies in similar contexts.

3.4 Theoretical model formulation

Sjøberg et al. ( 2008 ) proposed building software engineering theories in five steps: (1) defining the constructs of the theory, (2) defining the propositions of the theory, (3) providing explanations to justify the theory, (4) determining the scope of the theory, and (5) testing the theory through empirical research. These steps are not usually applied sequentially but are often carried out iteratively and partly in parallel instead.

In our theoretical model, constructs and propositions (steps 1 and 2) were extracted from our results as it is described in Sections 4.1.1 and 4.1.2 , respectively. In order to offer explanations for each proposition (step 3), we provide logical justifications based on interpretations of our empirical multiple case study (in Section 4.1.2 ), which is one of the ways explicitly mentioned in Sjøberg et al. ( 2008 ) to perform this task. Regarding the determination of the scope of the theoretical model (step 4), we specify all the contextual factors that define the cases for which the proposed theoretical model applies as attributes of the constructs. Finally, the testing through empirical research of the theoretical model (step 5) has not been done in cases other than the four cases of the present study. Further empirical research should consolidate, extend, or refute the results of the proposed theoretical model.

In this section, we present the findings from each case study and the results of the cross-case analysis. We first give an overview of the QR management strategies ( a plan containing different practices to handle QRs ) and practices ( activities that are carried out to handle QRs in order to realize the plan ) in each case company in Section 4.1 , with a cross-case comparison in Section 4.1.5 to answer Research Question 1. Then, we present findings regarding challenges in Section 4.2 to answer Research Question 2. We explain the differences in challenges and practices with the help of the different contexts of the companies in Section 4.3 to answer Research Question 3. Finally, in Section 4.4 , we present the findings more systematically in the form of the theoretical model of QR management challenges and practices. In the following text, all quotations from the interviews are in italics.

4.1 Strategies and practices to manage QRs in ASD

Overall, we found that the term QR is a fuzzy concept among practitioners in agile companies. When we asked them about QRs, the discussion tended to focus on quality in general and how it was managed. When seeking to discuss specific RE practices, the discussion tended to focus more on FRs. The interviewees naturally tried to reflect on the practices they were following, which were mostly related to FRs or to the development of features. The same phenomenon was observed regarding the questions about challenges. The interviewees tended to focus on the challenges the interviewees were currently facing in their work, which did not necessarily relate to QRs. The results reported in this section are relevant for QRs.

We characterized the strategies the companies apply as proactive, reactive, and interactive. Table 4 explains these strategies together with examples. The overall strategies applied by each case company mostly feature all of the above characteristics. However, one strategy of QR management can be more dominant than others. We do not intend to classify the strategies as one being better than the other.

4.1.1 QR management in company A

The goal of Company A is to deliver as stable and reliable a product as possible. Thus, the most prominent QRs are stability and reliability. The strategy they apply to reach this goal is mostly reactive and interactive. The only proactive characteristic is that the company roadmap, which contains the company’s long-term and short-term goals, is the first source for high-level QRs. The interactive characteristic of their QR management comes from the fact that the developers use the product in the development themselves and can identify concerns regarding the quality of the product. They allow the QRs to evolve during development. “ We are using the tool to develop the tool. That way, the developers can spot any problems. ” The reactive characteristic comes from the practice of not specifying QRs up front and relying heavily on feedback from quality assurance (QA) and users. This is possible because the product is mature and the most important QRs have already been established.

Instead of explicitly specifying QRs, the company relies on the experience and expertise of the developers to consider these QRs in development. “ It is a trade-off. Do you spend time doing specification, or do you spend time developing? ” Another interviewee said: “ The developers know it. They know the QRs. ” As for FRs or the features in general, specification and documentation of QRs is minimal, as promoted in the Agile Manifesto (Fowler and Highsmith 2001 ). Company A relies mostly on Word documents, emails, and whiteboards in documentation and communication. “ The QRs are mostly in the heads of the managers, and they are communicated through emails and Word documents, ” one interviewee reported. Another interviewee stated that “the Word documents can be considered as backlogs.” When the executive board has decided on the priority of features to implement, there is a meeting in which the most suitable architecture is discussed. There is not much more discussion about QRs at this point, unless the feature to be implemented is completely new. In that case, they produce detailed documentation to explore the extent to which the new feature is compatible with existing features or technologies; however, there is still no specification or documentation explicitly for the QRs, even though they are communicated verbally.

In testing, the company utilize both automatic testing in the CI and manual testing to identify quality concerns, and finally, they utilize customer or user feedback extensively. Users can provide feedback through the open source community, and paying customers can utilize a hotline for reporting issues directly to the sales team. These are means to identify “ quality issues, ” so called because they are reported as issues and could be translated into QRs. For example, a report could say that an element of the interface does not scale when changing resolution. This issue could be translated into a QR by stating that all elements in the interface should scale when changing its resolution. The quality-related errors found in testing are discussed between the QA team and the product manager to decide on the priority of the issues. Here, the “ gravity of the issue ” (i.e., the criticality), as some interviewees call it, determines the priority. For example, when a user experiences a crash—i.e., when the software stops working, and the user reports the issue—it receives the highest priority. “ Blocking issue ”—i.e., when the software prevents the user from accomplishing a task—have the second-highest priority. The customer-reported issues are prioritized in the same way as the internally found issues, with the difference being that they always get the highest priority.

Lessons learned

Company goals and roadmaps can be used for proactive elicitation of high-level QRs

With a mature product, there is a reduced need to specify all QRs up front

When developers use the tool in development, they can interactively evolve QRs during development

Close connection to users enables valuable user feedback for reactive elicitation of QRs

CI and QA can also be utilized for reactively eliciting QRs

4.1.2 QR management in company B

The most emphasized goal in Company B is to deliver value to the customer: “ We always try to think, what is the value of this feature? ” QRs are important for this goal. The strategy for QR management in Company B is a combination of proactive, interactive, and reactive. It is proactive in the sense that, currently, it includes practices for QR management already in the initial steps before actual development. Initial roadmaps and high-level epics include QRs that may come from, for example, the regulatory needs of a governmental customer. The interactive part comes from a periodical review/check done by the company to ensure that the development is in line with the goals. The reactive part comes through ongoing test automation in the CI, separate security testing, and trial user tests.

QRs are prioritized already at an early phase when roadmaps and high-level epics are defined. The value to the customer is an important theme in the prioritization. The product manager prioritizes features and requirements together with POs. Features are stored in product backlogs using Jira. At the feature level, QRs are considered in the Definition of Done (DoD) as quality targets, i.e., target values for general quality-related aspects in exit criteria, e.g., testing should reach a certain code coverage. One example of how a QR is considered is that, at the feature level, it is specified that all stability tests need to have been run. These are done through ongoing automated tests. When the features are broken down to user stories and further to lower-level tasks, these user stories and tasks include details that are more technical, and these include QRs related to, for example, performance or power consumption. However, low-level DoDs do not contain quality targets, as in the feature level, but include only items to be done and to ensure that everything has been tested. It is only the higher-level DoD that contains quality targets. The stability and performance QRs are tested in automated ongoing tests in the CI.

As the value to customers is important, it is also considered during development. Company B has so-called value stream mapping (VSM) sessions during development in which they periodically check that the product is in line with the goals, and that they are producing benefits for the end user. “ We start testing the feature. Does this feature work? Is this feature in line with the goal? Does this feature produce any value? ” This also helps Company B to reprioritize QRs. In testing, Company B applies ongoing test automation for performance and stability, and basic acceptance testing is performed after every sprint until release. A separate testing group focuses on security testing only because security is of high importance in Company B. Usability is not tested until a trial phase, which is performed by trial users who are personnel at the same company.

When there are standards and regulatory needs, it is necessary to proactively plan QRs up front

Software internal QRs (stability and performance) can be ensured by extensive ongoing tests

With limited access to users, an internal VSM session can help to align QR priorities with goals and values

When security is important, it is advisable to have a team of experts in security for testing

QRs can be documented as part of DoD and acceptance criteria

4.1.3 QR management in company C

Company C applies a strategy that utilizes mostly proactive and reactive practices for the management of QRs or quality in general. As the system is large and complex, some level of hierarchy is necessary to coordinate many development teams. Most elicitation, analysis, and specifying of QRs is done up front before implementation together with FRs in the features. Extensive testing in a multi-level CI is practiced to ensure quality as a reactive practice, and review meetings are held periodically to address quality concerns for the future. One interactive practice is also utilized for managing QRs; developers can elicit QRs during development and suggest their inclusion into the backlog.

Development is feature-driven, and all requirements, including QRs, are derived from features. In the initial phase, the elicitation, analysis, and defining of features are done by product managers. Factors such as customers, business opportunities, and the organization’s roadmap are taken into consideration in the feature elicitation and analysis from which QRs are derived. In the feature elicitation and analysis, QRs are included as quality criteria for features, just like in Company B. Features are further divided into sub-features before they are split into tasks for development. The sub-features are prioritized by managers for the POs. Thereafter, POs and Scrum masters divide the sub-features into tasks for development teams. When the quality criteria are broken down for the developers, they are written down as part of the DoD as a list of certain practices to follow, certain guidelines to follow, and certain tests to run. Some of the items listed in the DoD relate to QRs—for example, unit testing to ensure robustness.

For the reactive part, extensive testing is practiced to verify that the product meets the desired quality as well as functionalities. Testing is done at several levels, depending on the duration of tests, to speed up the discovery of potential faults. Developers can perform unit tests test when compiling before they push the code to the CI. “ We have three different quality gates in the CI. More parts of the system are integrated in each gate. ” In other words, there are three levels of testing in the CI. In the case of one of the business lines, which is responsible for developing a common platform software, the most prominent QR in integration testing is interoperability with existing systems or legacy systems. Any interoperability issues that arise from testing are communicated back as faults. Lower-level integration-testing faults are communicated directly back to the developers, and the developers are responsible for carrying out root-cause analysis. Higher-level integration faults are communicated back to specific quality roles in a monthly quality meeting, who are responsible for root-cause analysis. These quality roles participate in meetings with managers, in which they present the analysis of the faults, collaboratively decide on actions to take to correct the faults, and agree upon actions to take to prevent these faults in the future. The result may be a process change or an item to put in the product backlog.

In Company C, there is also another practice used to manage quality. There is a small reservation for quality-related initiatives as a percentage of the effort of a sprint. This means that any developer can suggest including quality-related items into the backlog. These quality-related initiatives can regard both process and product improvement. For example, a developer may realize that a certain QR might be easier to ensure with a certain test that is currently not done. He or she can suggest including the implementation of the test as an item in the backlog. As another example, a developer might realize that it would be much easier to pinpoint the root cause of a fault if he or she would have an additional logging of an event in the log file that the system produces. The developer can then suggest including in the backlog, as a quality item, that log file X should incorporate the logs of event Y. The suggestions are inspected, after which management decides whether to include the new suggestion; if so, the suggestion is sent to the developers to implement. In a sense, this is a practice to elicit quality-related items, if not QRs. We call this practice quality reservation-based elicitation of QRs when referring to it later in the paper.

Distributed development of a large and complex system requires some up-front planning of QRs

When interoperability with legacy systems might be challenging to elicit up front, a multi-level CI could help

Knowledge of the experts, i.e., the developers, can be utilized to elicit QRs during development, and the QRs can be proposed for inclusion in the backlog

4.1.4 QR management in company D

The strategy to manage QRs is proactive, interactive, and reactive in Company D; however, in comparison to other cases, Company D utilizes more proactive and interactive practices. QRs in general play an important role in software development in Company D. The visual aspects of the system to be developed are an important topic of the initial scoping sessions together with the client. In these scoping sessions, mock-ups are utilized to present a possible solution for a given problem. The mock-ups are used to elicit requirements, including QRs. “ We present them the mock-up filled with some dummy data. Does it look right? Does it work as it should? Do you find all information you need? Does it respond as you expect? ” If agreement is reached, a contract is made that includes the acceptance criteria, and a second round of scoping sessions is held in which further QRs are elicited with the help of mock-ups. Initial mock-ups, together with notes on client feedback as well as user stories and epics drawn up in the scoping sessions, serve as a baseline for the following implementation process.

The PO is mainly responsible for the prioritization of requirements; however, the prioritization decision is made after consultation with the development team. These requirements are mainly functional, and QRs are included in the FRs. “ For example, the functionality should provide the user some information in tabular form. Then there is a question of how many and which fields do you want to be shown. In what order should they be displayed? How should the rows be organized in the table? Should we include paging, infinite scroll, and so on? ” These details are necessary to be able to provide the requested functionalities. These details are documented in a template that the PO utilizes for specifying some of the QRs. The interviewed PO stated that this template has been designed mainly to help his own work. This way, time is saved, as details are recorded and given directly to developers, avoiding situations in which a developer has to stop work to get additional details from the PO. With the template, the PO can communicate some of the QRs up front and is the main channel of communication of QRs at the beginning of a project. The template is also reusable in other projects.

In addition, QRs are communicated and clarified in the meetings between sprints. Daily stand-up meetings, in which the product owner also participates, are additionally used for discussing QRs or acceptance criteria, if necessary. Additionally, Company D has review meetings with the customers during implementation to check the progress of every sprint or every second sprint, depending on the contract. With the help of this strategy, which includes the client from the beginning to the end, it can be ensured that the customers get what they ask for, and that the result is of sufficient quality. Finally, some QRs are included as acceptance criteria. For example, it might be specified in the acceptance criteria that the user interface needs to be responsive at a certain resolution, or that the front-end needs to support certain versions of certain browsers, or that support for multiple languages is implemented and working. Some aspects of QRs are included in the DoD; however, the DoD is more generic. The acceptance criteria are more detailed. “ To meet the acceptance criteria is one part of the DoD. ” Meeting the acceptance criteria is mostly achieved through manual testing at the end of development.

When accessible, users should be utilized throughout development—in elicitation, analysis, specification, and validation of QRs

When accessible, users can be consulted during development in intermittent review meetings to verify already elicited QRs and to identify possibly missed QRs

Mock-ups and quick prototypes are valuable when eliciting QRs from the users, especially if usability is important

Templates are useful for eliciting important QRs that users might usually omit

Templates are useful for specifying and document QRs, and for communicating them to developers

4.1.5 Cross-case analysis: Comparison of QR management strategies and practices

This section compares similarities and differences in QR management strategies and practices in the case companies. Many differences are attributable to the companies’ contexts, which are further elaborated in Section 4.3 . Table 5 summarizes the strategies and practices we found in the case studies. The overall strategies are in one category alone and the practices we categorize according to RE activities in SWEBOK 3.0 (Bourque and Fairley 2014 ).

What is common to all the studied cases is that they have clear quality goals and manage quality in general. The QRs are tightly coupled with FRs and are included in the acceptance criteria; however, practitioners appear to have a much better understanding of how to manage FRs, since none of the companies have as well-defined a plan for the management of QRs as they do for the FRs. For any specific QRs, the companies’ strategy is to complement their development processes with practices specifically oriented toward reaching their quality goals—for instance, having a separate team for testing security or utilizing mock-ups when user experience is important.

All of the case companies utilize proactive practices in their strategies, albeit a bit differently. Companies A to C employ a company roadmap as the first high-level source of QRs. They elicit QRs from the high-level feature or epics. On the other hand, Company D, as their development is contract-based, works closely with the customers, and is able to utilize them directly for QR elicitation with the help of templates. Most of the companies also utilize interactive practices for the management of QRs. Company D works closely with the customers whenever possible, while Company B practices VSM sessions in which they check the results of development with the goals. Company A uses the tool to develop it, and thus developers can notice if any QR-related aspects are not right, while Company C utilizes the expertise of the developers by having them suggest QRs to include in the backlog. For reactive practices, validation was very similar in all companies. All companies include some QRs in the acceptance criteria, and some are included as items in the DoD. Additionally, all companies utilize CI, to which developers contribute on a daily basis. QRs are distributed in different artifacts rather than being considered as separate items in the product backlog (Table 5 ).

4.2 QR management challenges

In this section, we present the challenges in managing QRs in ASD as elicited in our interviews. We categorize the individual challenges according to the area of effect. In that way, the categories follow the same structure from SWEBOK 3.0 (Bourque and Fairley 2014 ) as in the previous section. However, we situated implementation as an additional area of effect, in a category called “Challenges in QR implementation.” The category related to specifying QRs was denoted as “Challenges in specifying QRs” instead of “Challenges in QRs specification” in order to avoid confusion with the QR specification artifact. An overview of the challenge categories and the companies can be found in Table 6 . After the table, we describe each category and show what kinds of challenges can be expected in each area. We explain the reasons for the challenges and what kinds of consequences these challenges may have. We also reflect on the findings of the QR management practices and elaborate on how these challenges can be mitigated.

None of the challenge categories apply for all companies, as listed in Table 6 . Likewise, none of the companies are experiencing challenges in all areas. The effect of the context is explored in Section 4.3 after the challenges have been presented.

4.2.1 Challenges in QR elicitation

Challenges related to the elicitation of QRs were reported in three of our case companies: B, C, and D. One of the companies emphasizes value to customers and users; however, in this company, they have limited access to them. “We have very limited information about their actual use scenarios and how they use them, what they actually expect to get out of this product.” For this reason, they cannot be sure that they are getting the QRs right. For example, it might be deemed internally in the development organization that processing power is important to make all features work fast, when the context of use would not have required that, but battery power would have been more important. In such a scenario, there is risk of rework. In Company B, the practice of conducting VSM sessions during development, even though not helping elicitation directly, reduces the risk of rework.

In contrast, another company that does have access to customers and works closely with them also reported challenges in this area. At this company, part of the challenge came from the fact that customers, or users, rarely pay attention to software-internal QRs, like maintainability or scalability. For example, a customer may order a warehouse management system for one warehouse but fail to mention that, later, the system might need to support decentralized warehouses in different locations. Another example is language support for different languages. If the customer is a national company but fails to mention plans to expand to other countries, and thus, they may not think to ask for language support for different languages. This requirement is not purely internal, since it is also related to the usability of the system. Even though part of the challenge may be the fact that customers or users rarely pay attention to software-internal QRs, the interviewees reported the elicitation itself as a problem: “ How do you make sure that you get all the necessary QRs right from the beginning? ” It is many times the case that the PO does not have enough domain knowledge to be able to know all the important QRs. Even though Company D utilizes a template to collect details about QRs, the PO felt that elicitation techniques could be improved because there is always a risk that some important QRs might be missed and costly rework would be needed later: “ If you get everything right from the beginning, you just specify and develop. If you miss something, you do rework. ”

Another challenge related to the area of QR elicitation was the dependency between business lines, components, and development teams. One of the business lines in the biggest case company is producing a component for a common platform software. Products of the other business lines utilize this common platform software. Thus, because of the crosscutting nature of QRs, they have dependencies between different business lines, and interoperability between different components and legacy systems becomes a challenge. Interviewees stated that it is difficult to identify these dependencies: “ So, if Business Line X wants a change in our product, how do we make sure that that change does not have adverse effects on Business Line Y? ” Even though the challenge is different, and because of different reasons as in other companies, the consequence is the same—rework: “ Sometimes, when we have successfully satisfied a QR for one business line, it might come as a surprise at some point for another business line when they find out that something is not working any longer. ” The multi-level CI applied in the company helps to identify potential interoperability issues as early as possible.

Summary of challenges, consequences, and mitigation actions in QR elicitation

Not being able to elicit all important QRs

Consequence: Development focusing on wrong QRs

Mitigated in Company B later in development by VSM sessions

Customers or users might omit important software-internal QRs

Consequence: Rework

Mitigated in Company D by utilizing mock-ups and templates in elicitation

Identifying QRs with dependencies is challenging

Mitigated in Company C by multi-level CI to find interoperability issues early on

4.2.2 Challenges in QR analysis

This category includes challenges related to QR analysis. For example, it might be difficult to see the relationship between a specific QR and other requirements, whether a QR is process or product-related, or how to give the QRs proper priority. One of the challenges in this category is—and this is also related to the fact that practitioners find QRs to be a fuzzy concept—that “ it is difficult for us to measure quality. ” Many QRs are difficult to concretize and are more elusive and difficult to measure. When QRs are not easy to analyze and define in detail, they are accordingly not easy to measure; and consequently, they become difficult to specify.

Limited access to customers or users carries over to this area as well. Even if QRs are identified, it is still a challenge to identify what QRs bring the most value to the users; i.e., the challenge is to identify the right QRs to focus on. This makes the prioritization of QRs challenging, and the consequence is that sometimes development might focus on the wrong requirements. For example, if it is not known which communication bands the potential users would need or use, one might implement the use of many different bands, of which some would be unnecessary. One interviewee stated that the goal is to produce the minimum viable product as fast as possible; however, as in the example of communication bands, they would have already gone beyond the minimum viable product. Thus, they can feel that time has been wasted.

Dependencies pose a challenge for analysis too. Even if a dependency is identified, it is difficult to identify the exact effects of addressing that dependency on another component. If one QR is addressed from the point of view of one specific component, it might have adverse consequences on another component.

In our interviews, we found another challenge that also had a big impact on prioritization. This challenge was related to QRs competing for prioritization against FRs. Interviewees in one company said that customer features, and thus the FRs, get more priority than software-internal QRs. Interviewees stated that “ you need to give a good reason why to develop something quality-related, ” “ it is difficult to convince product managers why they should focus on quality, ” and “ it is hard to show the value of QRs in the backlog. ”

Summary of challenges, consequences, and mitigation actions in QR analysis

Challenging to know what to measure

Consequence: Unable to identify and know how to monitor and control quality

No mitigation practice identified from the practices of the companies of this study

Prioritizing QRs is challenging

Mitigated in Company B in VSM sessions by reflecting QRs to goals and targets

Identifying QR dependencies across components is challenging

Consequence: Not knowing the effect of a QR on all components

Mitigated in Company C by multi-level CI to identify dependencies

Focus on delivering functionality to customers is a challenge for QRs

Consequence: QRs get less priority compared to FRs

No mitigation practices identified from the practices of the companies of this study

4.2.3 Challenges in specifying QRs

We found only one challenge that we can categorize as a challenge for the activity of specifying QRs. One company specifies requirements in user stories and tries to also include QRs in those user stories. However, interviewees felt that these user stories are still many times too technical. “ It should not be a technical explanation that says, ‘add a button here.’ There needs to be the ‘why’ and the value behind that. ” One consequence that was mentioned, of not having enough background information or detail about a QR in the user story, was that even though functionality—i.e., what needs to happen—is described in detail, it may not say how . It might be that there are two ways to implement the “what” part, but only one is preferred. The “why” part could help identify the right “how” part.

Summary of challenges, consequences, and mitigation actions in specifying of QRs

Use of user stories for specifying requirements is challenging for specifying QRs

Consequence: QRs are not easy to specify in user stories and unclear QR specification

4.2.4 Challenges in QR implementation

Most of the individual challenges that were reported by interviewees are related to implementation. Some of these challenges might be seen as belonging to the previous category, Challenges in specifying QRs; however, they were not reported as challenges while specifying QRs. The results of the activity of specifying requirements caused challenges in implementation.

One interviewee in one company reported that “ granularity is a challenge; the specification is not detailed enough. ” That is, specifications are unclear or not always detailed enough for the developers to easily implement. Similarly, in another company, the unclear specification was reported as a challenge several times. However, in this company, the main reason was in some cases that specification was done by a separate group of people without the involvement of developers. Interviewees reported that “ specification is done without collaboration with developers ” and “ specification people and developers are not working together. ”

The developers also expressed that sometimes they get the feeling of “ are we even doing the right thing? ” Contributing to this was that the developers did not have visibility of the original requirements. Interviewees reported that “ the big picture is missing ” and that there was “ no visibility to customers ” in some of the business lines. Because developers have limited visibility to the customers or the original requirements, the developer expects that the specification is precise and unambiguous. However, developers explained that sometimes the specification leaves room for interpretation. The developer might have several options for how to implement a functionality, but how the developer ultimately implements this functionality may have architectural implications or might affect performance. This is why the developers feel it is important that they see the original requirements or the big picture. As a result, when developers are not involved in carrying out the specification and when they have limited visibility of the original requirements—i.e., they do not see the “why” part—they might not understand the specification as intended. This can lead to an issue whereby the development is focusing on the wrong things or, once again, to more rework later on.

Summary of challenges, consequences, and mitigation actions in QR implementation

Unclear or insufficiently detailed specification of QRs leaves room for interpretation

Consequence: Implementation of QRs becomes difficult

Consequence: Focusing on wrong QRs

Consequence: Development is slowed down

Mitigated in Companies B and D by including developers in the activity of specifying QRs

Mitigated in Company A by increasing developers’ visibility to the original requirements

4.2.5 Challenges in QR validation

Interviewees in two of the companies reported challenges that can be considered relevant in the validation of QRs. Again, limited access to users is the cause for one of these challenges. Because of this limitation, one interviewee reported that they receive feedback about QRs too late. Although one company practices VSM sessions and tests the product with internal trial users, the interviewee felt that they cannot be absolutely sure that they have been able to provide all the value a real user would want. The internal trial user can never replace the real end-user because the trial user does not operate the product under the same conditions. Additionally, the internal trial user might be biased.

Other reported challenges were “ limited traceability to the big picture ” and “ some QRs seen first on high-level integration testing. ” The first of these challenges is caused by developers having limited visibility of the original requirements during implementation. In a big organization, when the requirements are split into smaller parts and distributed to many teams, and when the intra-organizational distances start to grow, there is the risk that traceability will be lost. This is related not only to QRs but to any requirements. When a fault is found, the root cause analysis will be slowed down if traceability is lost.

The second challenge is related to the complexity of the dependencies of large complex systems. Even though a multi-level advanced CI system may find faults related to dependencies as early as possible, some dependency issues might still not be discovered until late system testing. When this happens, it produces an additional challenge. Testing a large and complex system takes a long time, and finding the right root cause for a fault might also be time-consuming. During that time, a team of developers might have already moved on to the new tasks in the following sprints; and since teams are dynamic—i.e., they are put together based on needed competences—it might be difficult to find the developer who was developing the part of the component that caused the fault. “ It is difficult to find the responsible guy much later after the sprint has ended, ” one interviewee stated.

Summary of challenges, consequences, and mitigation actions

Too late feedback regarding QRs

Consequence: Focus on wrong QRs, or QRs important to the user go unnoticed

Mitigated in Company B by VSM sessions

Mitigated in Company B by internal trial users

Maintaining traceability of QR-related test cases to design artifacts and requirements is challenging

Consequence: Finding root cause for faults may take a long time

4.2.6 General QR challenges

The challenges included in this section have a more holistic effect on development, not just on specific parts. They are more general, affect several areas of development, and are context-independent. Two of these reported challenges are related to the tendency of development to focus more on functionality than on the cost of QRs. Interviewees of one company reported that “ because of up-front planning and in case of tight schedules, quality gets less attention, ” and “ the improvement frame has decreased over time. ” Both challenges have effects on several areas of development. The reservation for quality-related improvements in the backlog was in place in one of the companies to enable developers to elicit QRs with regard to, for example, maintainability. However, it was stated by interviewees that these QRs have a limited possibility of getting enough priority and being included in the product backlog. If the QR in question would be related to maintainability, it would have severe effects later. The final challenge included in this category was reported as “ QRs are generally more challenging than FRs. ” This is included here, even though explicitly mentioned in only one interview, because the notion the interviewees had about QRs suggested that QRs are many times difficult to grasp. This confirms that practitioners find the QR to be a fuzzy concept.

Summary of challenges and consequences, and mitigation actions

More focus on FRs than on QRs

Consequence: Challenges in all areas of development

QRs are a fuzzy concept

Consequence: Challenges in all areas of software development

4.3 Context

We will now inspect the differences in QR management and related challenges and explain these differences with the help of the different contexts of the companies. We started by studying which areas in the companies faced challenges, and what kinds of challenges were faced. Next, we studied the differences in the challenges and the contextual elements of the companies in order to determine whether the contextual elements could explain why some companies experience challenges while others do not. When we found a specific contextual element causing a certain challenge, we realized that the same element could also work as a driver for applying a certain practice. That way, we could explain why a specific company was not experiencing a certain challenge while another company was. This led us to map contextual elements to different practices applied in the management of QRs. This way, we identified nine different contextual elements from the four different context facets that may affect the choice of practices or the challenges related to QR management in ASD. From the product facet, the elements were product type, maturity, and customization. From the process facet, the elements were the characteristics of the development process, whether or not development was distributed, and the number of employees involved in the development of the product or component. From the organization facet, the element was whether the organizational model was hierarchical. From the market facet, the element was the type of customers or users and access to customers. Tables 7 , 8 , 9 and 10 summarizes the contextual elements driving the practices that explain the challenges which are summarized in Tables 11 , 12 , 13 , 14 , 15 and 16 .

4.3.1 Context and QR management practices

In this section, we start by inspecting the management strategies and practices per RE activity as defined in SWEBOK 3.0 (Bourque and Fairley 2014 ) and in Table 6 . We identified seven different contextual elements affecting the choice of strategy and practices: product type, maturity of the product, customization, characteristics of the development process, distributed development, hierarchical organizational structure, and access to customers or users.

Elicitation of QRs

All the companies apply a proactive approach and try to elicit QRs up front before implementation, although differently. Company D utilizes the users of the product for QR elicitation. Ultimately, there will be users using the system whose productivity might depend on the system. For this reason, they need to work closely with the users to develop the right solution. As such, the product and market facets of the context affect the choice of practice in QR elicitation. This would naturally also require that there is access to the users of the product. This is not the case in Company B, which relies on company goals and roadmaps for the product to elicit important QRs. Also, Company A has access to the users; however, the company does not interact with users in the same manner as Company D. The product that Company A develops is a mature tool that already has an established set of requirements. Therefore, the company does not have the same need to elicit specific QRs. The users are more valuable in giving feedback about how the product performs and compares with other, similar tools. The company uses this feedback as a technique to elicit additional QRs. Additionally, the company has adopted a development mode that is highly reactive to this feedback. This can also be the reason why Company A does not experience any challenges in the elicitation of QRs. Company C is producing a large system with complex dependencies. It is challenging for the company to elicit all the QRs that are affected by the dependencies of different components up front, so as an aid it utilizes the multi-level advanced CI to do so.

Analysis of QRs

The situation in the analysis of QRs is very similar to elicitation, and for the same reasons. Company D has access to customers or users and utilizes them to analyze the QRs together with the customer using mock-ups and reusable templates. Company B does not do this because of limited access to users, and Company A does not experience the need to do it together with the users due to the maturity of the product. In Company A, there is also another element of the product facet that affects the choice of not analyzing all QRs to the smallest detail: the customization element. The product is the same for all users. This means that the product can be installed in a great variety of environments, and it is virtually impossible to identify all of those environments. Thus, it is easier to address, for example, the main stability issues and address other issues as soon as they emerge. Additionally, the developers use the product to develop it. Company B, which has limited access to users, practices VSM sessions to analyze QRs. All companies evolve the QRs together with FRs during development by analyzing and reviewing the QRs further in daily and weekly meetings, although in Company C this practice is somewhat limited because all requirements are analyzed and specified by a separate internal organization. It could be said that the hierarchical organizational model and distributed development together dictate this choice.

Specifying QRs

Many of the interviewees stated that they did not specify QRs; however, some of the QRs are still specified in the acceptance criteria, while others are specified as quality-related items and are included in the DoD. Interviewees in Company A reported that they do not exercise extensive specification of QRs, but they instead rely on developers’ experience and expertise to consider QRs during implementation. The same applies for Companies B and C. Company B tries to include QRs in user stories but reports doing so as challenging. Company C, on the other hand, aims to conduct the most extensive up-front specifications. The nature of the distributed development affects this choice, since QRs can influence the development of several teams. Company D utilizes a template for specification and documentation of some of the QRs up front and communicates them to developers in this template. However, the company also elaborates QRs further during development. It could be said that most companies avoid specifying QRs too heavily up front, which is in accordance with agile methods. Thus, the characteristic of the development process would influence this strategy.

Validation of QRs

CI and acceptance criteria seem to be the chosen practice to ensure the quality of the product in all companies. Multi-level CI appears to be especially useful with large and complex systems. In addition to testing, Companies A and D utilize the users in validation as well. As explained earlier, Company A utilizes user feedback to correct any QR issues. Although this can be seen as a very late reactive practice, since it happens after release, it is less resource-demanding, and the company has adopted a highly reactive development process. In Company D, review sessions are conducted together with the customers or the users, and thus any QR issues can be corrected before release. The contextual elements of access to customers or users and customization explain this difference. Company D is developing a custom solution for a specific problem, while Company A is producing a general product that can be used in many situations and diverse environments. In the case of Company A, it would be an overwhelming task to attempt to address all the contexts of use of the tool to identify all possible quality issues. Thus, it makes more sense for the company to instead react to quality issues found by users as fast as possible after release. If it is not possible to get the final end-users to test the product, then the development organization should have a group of internal people test the product, as is done in Company B.

4.3.2 Context and challenges

In this section, we explore the connection between context and the identified challenges in each area of effect and aim to elucidate how the contextual elements may cause challenges. We use the same grouping of challenges as in Table 6 . We found five contextual elements that cause challenges in the different areas: access to customers or users, hierarchical organizational model, characteristics of the development process, and type of product.

Contextual elements and challenges in QR elicitation

From the interviews, we could identify two contextual elements that cause challenges in QR elicitation. One such challenge comes from the constraint of the market facet, i.e., limited access to customers or users. This has a negative effect on elicitation. If it is difficult to elicit the right QRs together with the customers or users, it is even more so without access to the customers. The other contextual element that causes challenges is the system with complex dependencies of the product-type facet. It is not always easy to see all dependencies up front. Additionally, we found the challenge of how to get all important QRs from the users. However, we do not believe that this challenge is caused by any contextual element but is rather a more general challenge in the elicitation of requirements.

Contextual elements and challenges in QR analysis

Two contextual elements cause challenges in QR analysis. The challenge of identifying what is most valuable to customers, which affects the prioritization of QRs, is derived from limited access to customers or users in one case. The other contextual element is the system with complex dependencies. For example, the interoperability QR can be known, but what is not known is exactly how different parts of the product will be affected, and it is thus difficult to analyze dependencies between the tasks of different teams. Additionally, there is one challenge affecting QR analysis that cannot be mapped with a contextual element: QR is a fuzzy concept. Because of this, some QRs can be difficult to concretize and measure. This is, however, a more general challenge.

Contextual elements and challenges in specifying QRs.

Regarding the activity of specifying QRs, we found that the only challenge is the difficulty of including QRs in user stories. User stories are widely used in agile methods. Thus, we can say that this challenge originates from the contextual element of the characteristics of the development process. For this challenge, we did not find an effective mitigating practice. The interviewees themselves suggested that user stories should include additional information about QRs. How user stories could be extended or enhanced to also include QRs is a topic for further research.

Contextual elements and challenges in QR implementation

Most of the challenges were related to implementation. One reported challenge was that specification is not detailed enough. In this particular company, it can be said that the challenges are derived from the characteristics of the development process. The company has chosen not to spend much time on specifying or documenting the QRs. This is in line with the Agile Manifesto (Fowler and Highsmith 2001 ). In another company, most of the challenges in implementation, like unclear specification, misunderstandings of specification, and the missing “big picture,” are also rooted in the activity of specifying the QRs. However, in this company, the requirements were specified without the involvement of developers. Thus, it can be said that this challenge comes from the hierarchical organizational model. The distributed development and plan-driven characteristics of the development process also contribute to these challenges. Requirements are specified up front, and the developers are detached from the original requirements.

Contextual elements and challenges in QR validation

The challenges in validation in one of the companies are once again related to limited access to users. Interviewees stated that they received feedback from users quite late, and that they would like earlier user feedback. In addition, the same contextual elements of hierarchical organizational model, distributed development, and characteristics of the development process also contribute to the challenges in validation. Final system-level integration is accomplished late from the perspective of a development team in one company. Because of this, a development team may obtain feedback from integration testing long after the related sprint has ended. At that time, root-cause analysis might take a long time, and it might be difficult to find the developer who was developing a specific part of a component.

Contextual elements and general QR challenges

Two of the challenges in this category were related to the focus of development on functional features at the cost of focus on QRs. It could be said that this is inherited from agile development and thus from the characteristics of the development process contextual element. However, in one of our cases, the contextual element of type of customer also contributed to the focus on functionality. If the customers are company-internal—that is, the affected users are developers and engineers—QRs might receive less priority if end-user functionality is not directly affected. The other reported challenge was that QRs are generally more difficult to manage. This challenge has no connection to context. To raise the awareness of quality and QRs among all stakeholders, it could be helpful to shift the focus more toward QRs than functionalities alone.

4.4 Theoretical model

This section presents the results and analysis from our multiple case study in the form of a theoretical model of QR management challenges and practices in ASD. The model was designed to systemize the findings obtained from the present study. For this purpose, we used the methodology proposed by Sjøberg et al. ( 2008 ), which was developed to generate theories that can explain or predict phenomena occurring in software engineering. Our model can be used as the basis for explaining the relationship between contextual elements and QR management challenges, and it can thus be used to predict the challenges that companies can face in certain contexts. Further research may consolidate the findings of the present study, yield new findings, or refute some of our propositions.

According to Sjøberg et al. ( 2008 ), a theory is formed by introducing its main constructs and then establishing propositions as relationships among them. Constructs can be (1) classes defined as specializations or components of one of the four archetype classes, namely Actor, Technology, Activity, and Software System, or (2) attributes inside these classes. Archetype classes are predefined in Sjøberg et al.’s approach; their purpose is to provide a uniform terminology and structure to formulate theories in software engineering. Propositions relate a source and a target element. The source of a proposition can be a class or an attribute, and the target can be an attribute or another proposition. When the target of a proposition is another proposition, this means that the source element affects the direction and/or strength of the effect of the target proposition.

Before describing the theoretical model, we illustrate its usefulness by means of proposition examples that belong to the resulting theoretical model. On the one hand, the propositions are able to express the relationships between contextual elements and QR management challenges. For instance, the proposition “Limited access to users increases the difficulty of eliciting QRs in ASD” relates a source contextual element, i.e., the limited access to users of the development team actor, and a target element representing a challenge, i.e., the difficulty of the elicitation activity of the QRs. Furthermore, propositions can also relate practices to the challenges they face. For instance, the previous proposition is itself the target of another proposition, meaning that there exists a source element that affects its strength—that is: “Use of roadmaps and value goals for QR elicitation decreases the difficulty of eliciting QRs when there is limited access to users in ASD.”

The rest of the section presents a description of the whole set of constructs and propositions of the theoretical model.

4.4.1 Constructs

In this subsection, we present the constructs that we have defined for each of the predefined archetype classes and illustrate them graphically in Fig.  4 .

A theoretical model for QR management in ASD: constructs, general propositions (solid arrows), and their mitigation propositions (dashed arrows)

The diagram shown in Fig. 4 follows a UML-based notation proposed by Sjøberg et al. ( 2008 ). Classes are drawn as boxes. Specializations between classes are denoted by the UML generalization arrow, e.g., Elicitation practice is a subclass of QR management practice . Component classes are depicted as boxes within another box, e.g., Project is a component of Organization . Attributes are textually represented inside their corresponding classes. As already done above, terms that appear in the model are written in italics in the text. The arrows in Fig. 4 represent propositions and will be presented in Section 4.4.2 .

Software system : This class represents the scope of the study. In our case, we are interested in a specific type of system, i.e., software systems developed using ASD ( agile-developed software system ). A specific agile-developed software system should satisfy certain QRs that correspond to a QR type . The software system is involved in a software product (e.g., an embedded system, a software tool) that has a certain type (e.g., large with complex dependencies), a maturity level (e.g., a long-lived mature product, a new product) and a customization degree.

Technology : This class represents the focus of the study. Following Sjøberg et al. ( 2008 ), this archetype class must be interpreted in a broad sense, and the focus of the study can range from a process model, method, or technique to a tool or language. In our case, the focus is the QR management process in ASD . To understand how QRs are managed in a particular context, it is necessary to consider:

The contextual software development process that is being followed. From a particular process, it considers the method taken as a basis ( base method , e.g., Scrum, Kanban), its release frequency , the feature language used (e.g., natural language), and its feature management artifacts (e.g., product backlog).

The QR management practices that are being applied. A QR management practice is classified as an elicitation practice , specification practice , analysis practice , or validation practice . The aspects to be considered in each practice appear as attributes in the corresponding class.

Actor : The main actor in the theoretical model is the organization that manages the development of the software system. According to our study, the context attributes to consider in a particular organization are size , domain , geographical distribution , organizational model , and market setting (e.g., business-to-user, niche market, business-to-business). In an organization, we identify as crucial actors the project and team . For a project, we consider the type of customer (e.g., national organizations, software developers). For a team, we consider its size , whether it is distributed or not, whether it has access to users or not, and the understanding of QRs of its members.

Activity : The main activity studied is the development of a software system ( develop software system ). The important sub-activities of this main activity are the following: elicit QRs , specify QRs , analyze QRs , implement QRs , and validate QRs . For each of these activities, it is relevant to know about its difficulty ; and for QR specification, the clarity of its results is also relevant.

4.4.2 Propositions

The propositions obtained from our multiple case study can be classified as general, context, or mitigation propositions. General propositions are those that specify relationships not dependent on contextual aspects and, hence, they are susceptible to appear in any context (e.g., “In ASD, there is no shared understanding of the concept of QR”). Context propositions state relationships in which contextual elements are involved (e.g., “Limited access to users increases the difficulty of eliciting QRs in ASD” is a proposition about contexts in which the team has limited access to users). Finally, mitigation propositions specify cases of elements that can mitigate the negative effects stated by the previous general or context propositions (e.g., “Use of roadmaps and value goals for QR elicitation decreases the difficulty of eliciting QRs when there is limited access to users in ASD” is a proposition about mitigating the negative effects on QR elicitation of limited access to users).

The propositions can be graphically represented, following Sjøberg et al.’s ( 2008 ) notation, by means of arrows that connect their sources and targets. For readability purposes, we have distributed the representation of our propositions in two figures. Fig.  4 illustrates general propositions (solid arrows) and their corresponding mitigation propositions (dashed arrows), while Fig.  5 illustrates context propositions (solid arrows) and their corresponding mitigation propositions (dashed arrows). The constructs are the same in both figures. The definition of the whole set of propositions is presented in Table 17 . Table 17 also provides, for each proposition, an explanation that justifies it in terms of the results of the study. General, context, and mitigation propositions are referred to as G k , C k, and M k , respectively, and their corresponding explanations as EG k , EC k, and EM k , respectively.

A theoretical model for QR management in ASD: constructs, context propositions (solid arrows), and their mitigation propositions (dashed arrows)

5 Discussion

In this section, we discuss the key findings and compare them with the scientific literature. Afterward, based on the findings, we discuss the implications for researchers and practitioners alike. Finally, we discuss the limitations of our research.

5.1 Key findings and reflections on the previous literature

We divided this section into three subsections—practices, challenges, and context—for reflecting on our findings in light of the previous literature.

5.1.1 Challenges

In our study, the challenges stated by interviewees were grouped according to area of effect in order to help connect the contextual factors rather than classifying them according to themes of the challenges. Still, we could see that most of the challenges that we found can also be found in the scientific literature. Some of these challenges are: QR is a fuzzy concept among practitioners (e.g., Alsaqaf et al. 2019 ), users do not focus on software-internal QRs (e.g., Aljallabi and Mansour 2015 ), user stories are challenging for QRs (Rodríguez et al. 2009 ), and feedback regarding QRs is obtained too late (Alsaqaf et al. 2019 ). A more comprehensive list of all the QR-related challenges reported in the scientific literature can be found in our SMS (Behutiye et al. 2019 ), and as such there is no need to repeat them here.

Not all of the challenges found in literature were identified in our study, nor did we find any new challenges to add. However, we were able to provide more details for some of the challenges presented in literature to help understand the challenges better. From what can be found in the literature, and from what we did not find, we can point out a few examples. In the literature, it is reported that QR implementation many times relies on tacit knowledge (e.g., Heikkilä et al. 2015 ). This was the case in some the companies in our study, to some extent; however, it was not explicitly stated as a challenge. Instead, some interviewees said that in the case of unclear specification, they would have liked to have had visibility of the original requirements to help guide them in the implementation of QRs. This can be limited in companies in which developers are not involved in specifying requirements. Additionally, relying on tacit knowledge can be problematic for software development in the domains of safety and security (Knauss et al. 2017 ). Although one of our companies used Scrum almost by the book, and safety and security were major concerns for this company, it did not adopt any structured framework, such as Safe Scrum (Stålhane et al. 2012 ) but instead complemented the Scrum process with additional documentation for security and safety and a separate team specialized in testing security. The use of user stories is considered a strength in ASD, but it is difficult to include QRs in user stories (Alsaqaf et al. 2017 ; Savolainen et al. 2010 ; Bourimi and Kesdogan 2013 ). Three of our cases utilized user stories, but the challenge was reported by only one of them.

What is more interesting is that some of the QR management challenges found in our study are very specific to the companies and are often not a challenge in ASD only. For example, one stated challenge was limited access to user scenarios. This was due to the company’s attempt to enter a niche market in which it had not yet established good connections with the market’s users. The challenge in this regard would have been present irrespective of the mode of development. Complex dependencies across different components of a large system was another company-specific challenge that is directly related to the complexity of the system under development. This complexity would be present in any kind of development. In fact, it might be argued that this particular challenge is less so in ASD that applies CI since less effort is spent up front on something that is very difficult to find or specify up front.

Other interesting points are that the challenges we found were relatively scarce—for example, challenges in the specification of QRs came only from one company; in Company D, interviewees mentioned challenges only in QR elicitation. We did, however, identify the general challenge that QRs are fuzzy, as also found in literature (Alsaqaf et al. 2017 ), and it is not easy to know what to measure regarding some QRs. Should this be the case, then, consequently, it would be difficult to specify QRs. One reason, at least in one company, could be the fact that it simply chose not to specify QRs in detail. In Company A, the most important QRs of the mature product had already been established, so for this company it was not cost-effective to expend efforts up front on trying to specify something about which it was uncertain. Instead, the company relied on CI and usage feedback. On the other hand, most companies claimed that they specify QRs in DoD and acceptance criteria. Specifying QRs in the acceptance criteria is part of Requirements Specification for Developers (RSD) approach, which has been evaluated in small companies, and which includes an extended acceptance criteria targeted for software-internal requirements (Medeiros et al. 2020 ). The approach also includes the use of mock-ups. In Company D, the mock-ups and templates were utilized for elicitation, analysis, and specification of QRs. With the help of the mock-ups and templates, the company actively elicited internal QRs in such detail that they could be specified for developers. In that sense, the practice is similar to RSD. In the largest of our companies, Company C, specification was not always best suited for developers. Perhaps even large-scale development could benefit from the RSD approach.

Not finding as many challenges as would be expected based on the scientific literature and on the number of interviewees may imply that companies applying ASD are becoming increasingly quality aware, and that they are already finding ways to manage QRs. This could also be supported by the finding that many challenges can be seen as non-specific for ASD. Those QR management challenges that were previously ASD-specific are perhaps gradually being resolved, while those that remain are QR management challenges that depend purely on context or that have always been an issue—for example, how to collect all relevant QRs. In cases of large systems with complex dependencies in which some level of hierarchy is need for decision making, there is an increased general need to pay careful attention to aspects such as documentation, communication, and traceability, as well as to generally and carefully align RE with testing through people, processes, and artifacts. This alignment has been the interest of many large software development organizations in the automotive and avionic domains (Karhapää et al. 2017 ), and QR management could benefit from it.

Since Alsaqaf et al. ( 2019 ) is the most recent study, and since one of our companies operates in large-scale ASD, a brief comparison is warranted. We can see similarities in findings regarding QR challenges, however, we have treated these findings somewhat differently in our studies. For example, one of the companies in our study stated that “users do not pay attention to software-internal QRs,” highlighting this as a challenge for which the company had developed a mitigation practice. However, in the study by Alsaqaf et al., “customers are not interested in internal QRs” was identified as a mechanism behind “QR elicitation challenges” rather than as a challenge on its own. Still, we can confirm that users’ interest in internal QRs is limited. Table 18 lists challenges found by Alsaqaf et al. ( 2019 ) and we compare them to our findings and point out similarities.

As can be seen from Table 18 , there are similarities but also differences in findings regarding challenges. In addition, it should be noted that not all the challenges that are similar to those of Asaqaf et al. were found in the largest of the case companies of our study. The challenge of too late feedback about QRs was stated in one of the larger SMEs and users limited interest in internal QRs in the smallest SME. One of the most evident dissimilarities concerns the QRs documentation: Alsaqaf et al. found the challenge, “confusion about QR specification approach,” and discussed, based on this, the possibility of abandoning the concept of requirement in agile development. According to the authors, for agile practitioners, the concept of requirement included “the user story + the conversation about what is in the user stories + the acceptance criteria” (Alsaqaf et al. 2019 , p. 50) and suggested that using this variety of documentation strategy may introduce further inconsistencies into the development process. In our study, the QRs were often included in the acceptance criteria and DoD, and the interviewees did not find this to be problematic. It seemed natural for the practitioners in our study to include certain QRs in the acceptance criteria and others in the DoD.

5.1.2 Practices

In our own SMS about QR management (Behutiye et al. 2019 ), we identified 143 papers that examined practices, methods, models, frameworks, advice, tools, and guidelines for QR management. Out of these studies, 43 focused on practices, with a total of 74 practices of which many have not been validated in industry. In light of this large number of practices, one might expect that all practices that we found in our study have already been included in the literature—indeed, most of them have. For example, the practice of utilizing experts in the field of security can be found in Ayalew et al. ( 2013 ) and S. Türpe and Poller ( 2017 ); the practice of using mock-ups (or “low-fi prototypes”) for QRs is described in Wale-Kolade et al. ( 2014 ); the practice of continuous and automated monitoring and testing is discussed in Cruzes et al. ( 2017 ), Cannizzo et al. ( 2008 ), and Gary et al. ( 2011 ); and the practice of using acceptance criteria and DoD, among others, for documenting QRs is detailed in Behutiye et al. ( 2017 ). However, not all of these practices were used exactly as they were proposed in the literature, or for the exact same purpose. For example, Wale-Kolade et al. focused on usability, whereas Company A utilized mock-ups and templates to capture any kind of QRs. Cruzes et al. focused only on performance testing, and Gary et al. focused on security testing, yet CI was employed in all our case companies for ongoing testing to fulfill their respective QR needs. Company B utilized VSM sessions periodically, as they explained, to ensure that what they have developed so far was in line with quality goals. Käpyaho and Kauppinen ( 2015 ) proposed prototyping with a focus on reviewing the big picture at steady intervals and reviewing prototypes with a focus on QRs. However, Käpyaho and Kauppinen did not discuss the potential of this practice in the context of limited access to customers. As we have shown, this contextual factor could be the driver for the practice in Company B.

Since similarities do exist between the practices proposed in the scientific literature and the practices found in our case companies, it would seem that these practices have been adapted from the literature. However, it would seem that the sources for the practices were rather in their own efforts to solve challenges. For example, in Company D, it was explained that the use of the template for eliciting QRs and details of QRs was developed by themselves in an attempt to solve their needs. The only practice that we did not find in the scientific literature was “developers using the tool to develop it,” which was used by Company A. This practice is very specific for a certain type of situation—that is, when a tool is being developed that can be utilized in the development of the next version of that same tool. It is highly unlikely that this practice would be applicable in any other kind of scenario.

Alsaqaf et al. ( 2019 ) identified a set of practices in their empirical study. Just like the practices we identified in our study, those discussed by Alsaqaf et al. are very specific to the contexts of the companies they examined and are therefore not directly comparable. However, some similarities exist. In Table 19 , we reflect on the practices applied in our case companies with regard to the mitigation practices covered by Alaqaf et al. We compare most of the practices to Company C, since this company operates in large-scale ASD.

Similarities between the practices found by Alsaqaf et al. ( 2019 ) and those identified in our study can be evidenced through comparison. However, in our study, these practices did not always surface as practices for mitigating specific challenges. This is probably because the identified challenges are not the same. Likewise, we identified practices in Company C that were not reported by Alsaqaf et al. Company C applied a multi-level CI with different quality gates to catch, for example, internal QR issues as soon as possible. QRs documented in the acceptance criteria and DoD are reported as a practice in our study; however, in Alsaqaf et al., this was regarded as a challenge, one which caused confusion about the specification approach of QRs.

5.1.3 Context

When we grouped the challenges, compared them between different companies, and then compared the contexts of the companies, we could see that some of the contextual elements could explain some of the challenges and also work as drivers behind some of the practices. Likewise, context might explain why none of the identified challenges were universally experienced by all the studied companies. It might be that a specific challenge is not experienced because of the way of working, or because it did not have a major effect in a certain context. A comparison with existing literature, however, is very limited. In our SMS, we found that research into the interplay of context and QR management practices and challenges is almost completely lacking. Alsaqaf et al. ( 2019 ) is the only other study that we found that took context into consideration. They explored whether challenges vary in terms of the severity or risks they pose to companies. In their study, they related context mainly to domain. They listed seven business sectors: government, public transport, tax services, commercial/business, transport navigation, banking, and insurance. The type of system was either an information system or an embedded system. Ultimately, they did not find a relationship between context and challenges.

5.2 Implications for research and practice

From all the different challenges experienced by the case companies as well as the various practices employed to mitigate them, we were able to identify sets of practices for each area of software development that can be applied in ASD. Practitioners can use these results to guide their decision about QR management practices and help to solve challenges that ASD is prone to when managing QRs. In order to keep the text shorter, we included the list of practices in Appendix 4 Beneficial practices for QR management in ASD. As an example, Fig.  6 illustrates the practices that can be applied in the case of access/limited access to users. When there is limited access to users, both QR elicitation and QR validation can benefit from internal trial users while internal VSM session can help in analysis of QRs. When a company does have access to users, the users should be utilized throughout development. Mock-ups and templates can help QR elicitation, analysis, and specifying while reviews meetings together with the customers can help in QR validation.

Practices applied for QR management with limited access to users compared to full access to users

The relationship between context and QR management, or between context and QR management challenges, is complex and requires further study by researchers. More possibilities exist for drawing connections between contextual elements, chosen practices, and experienced challenges than those we have elucidated in the present study. Moreover, there are likely other contextual elements with similar effects that we did not discover in our study. Additionally, contextual elements could be used to explain why a certain practice is not used, even when it is apparently beneficial, as well as why a certain challenge is not experienced, even though it should be expected. Understanding all these connections would be of significant value in designing both general QR management guidelines and specific QR management processes for practitioners.

5.3 Validity discussion

We classified validity concerns into four different aspects of validity, as proposed by Runeson et al. ( 2012 ), in this multiple case study: construct validity, internal validity, external validity, and reliability.

In our study, construct validity related mostly to whether interviewers and interviewees possessed the same understanding of the interview questions. The interview questions were created by two researchers and reviewed by two other researchers. All interviews, except the complementary interviews conducted with Company C, were carried out by two researchers. While one researcher asked the interview questions, the other used a checklist to ensure that all necessary information was recorded. Since the researchers and practitioners did not necessarily use the same terminology when talking about a certain concept, preparations were made to explain the terminology whenever needed. For example, in some cases, the practitioners preferred to talk about, and understood better, the concept of NFRs instead of QRs. Still, there was always the possibility that an interviewee would not want to admit that a concept was unfamiliar. For this reason, we encouraged more open discussion instead of short and simple answers. There was also the risk that interviewees would not want to report on the challenges the company was currently facing. This was mitigated by assuring the interviewees that all information would remain anonymous and confidential, and that only a summary of the interviews would be presented to others. In some cases, we observed that some interviewees were very eager to point out challenges—and for this reason, some challenges might have been exaggerated. However, we did not focus on the severity of a challenge reported by one interviewee but instead on the frequency with which the challenge was reported. Additionally, the number of interviewees in the largest company was much larger than in the other companies. This approach was necessary for yielding a fuller understanding of how the different business lines operate. We acknowledge that this might skew the results toward making challenges or practices in the largest company more prominent. However, we did not prioritize challenges based on number of times mentioned, nor did we exclude challenges or practices mentioned by individual interviewees in smaller companies since we wanted to gather as many QR management related challenges and practices as possible.

Internal validity refers to the causal relationships between factors. Since we were investigating the relationship between context and QR management challenges and practices, internal validity had to be considered. One researcher conceptualized the connections between QR management practices and challenges and the contextual elements of the companies. Two other researchers reviewed the results and used this information to construct a model of the connections as well as to check whether these connections were realistic. As can be seen from our findings, not all challenges were related to the contextual elements. Thus, it can be assumed that factors other than those presented in this study can also affect the challenges. However, the aim was to find those challenges that were most evidently affected by contextual elements. Additionally, it is possible to think of QR type as a contextual factor in itself. Different types of QRs could require different kinds of management practices. For example, practices to manage usability related requirements rarely fit for security requirements, and vice versa. This aspect could be further explored in future work.

External validity must also be considered, since we sought to find connections between context and practices and challenges that others could utilize. All the studied companies operate in different context, thus, there is only one company with a specific context. Yet, some of the practices were applied in all the companies. These practices are typical for or favored in ASD, and thus it can be said that such practices are likely to be found in companies outside our study as well. Some practices are unique in a certain context, and as such it can be argued that they have emerged in response to special needs, or specific challenges, faced by a certain company. However, whether these practices could be effective in mitigating challenges in other companies with similar contexts and similar challenges is an issue in need of further research. None of the challenges were experienced universally by all the case companies. Thus, even though we were able to rationalize the connections between certain elements of the context and the challenges, of which some were obvious, these connections would need to be confirmed with further studies.

For reliability , both data collection and analysis were mostly done by peers. Two researchers independently devised coding schemes that were used to code the interview transcripts. Independent coding was done in a top-down and bottom-up fashion by using the same transcripts. The resulting coding schemes were compared and discussed to form a common understanding of how the transcripts were to be coded. The common coding scheme was then used on another transcript by both researchers, and the results were compared once again. At this point, there were only minor points of discussion, and we were thus able to use the coding scheme on the rest of the interviews. The top-down coding employed the same structure as the interview questions. CMMI for development, version 1.3 (CMMI Product Team 2010 ), was used in this regard. For this part, other researchers could arrive at the same coding scheme. However, the bottom-up coding is more subjective, and a different set of researchers could generate different results. The results were validated in each company in workshops by the company champions and other key actors who had knowledge about the software development processes. Additionally, the company champions reviewed this manuscript to validate that the included information was accurate.

6 Conclusion and future work

We conducted a multiple case study to investigate strategies and practices for quality requirement (QR) management in agile software development (ASD) and related challenges. Additionally, we identified contextual elements that contribute to the choice of practices and to the emergence of specific challenges. The case study was conducted in four different companies of different sizes and contexts through semi-structured interviews. We found that the practitioners in the studied cases manage quality in general and complement their development processes with any practices necessary for the management of QRs to reach their quality goals. We classified these practices as proactive, reactive, and interactive, and then grouped them as practices in QR elicitation, QR analysis, specifying QRs, and QR validation.

The challenges reported by the interviewees were different depending on the context of the studied companies. We grouped the challenges according to the area of effect as challenges in QR elicitation, QR analysis, specifying QRs, QR implementation, and QR validation. Some challenges were more general and affected several areas of development, i.e., they were context independent. Other practitioners can utilize our findings to identify what kinds of challenges they can expect to face in different contexts.

Many of the challenges reported in the literature were also found in our case companies, albeit with some slight variations. We also determined that not all challenges found in the scientific literature were present in our studied case companies.

We were able to identify a set of practices that have a positive effect on different areas of software development and can as such help mitigate some of the challenges we found. Practitioners could utilize these findings to improve their own QR management practices.

We identified many connections concerning how context drives the choice of QR management practices and how contextual elements can be the source for challenges. However, opportunities remain for the further exploration of these connections. For example, future research could examine why some practices are not used despite being apparently beneficial, or why some challenges are not experienced even though they should be expected to occur. An investigation into these connections would complement the findings of this study and ultimately complete the theoretical model on QR management in ASD. In addition, analyzing whether there are differences in managing QRs depending on the type of QRs and, if so, what those differences are would be interesting.

http://q-rapids.eu/

Aljallabi BM, Mansour A (2015) Enhancement approach for non-functional requirements analysis in agile environment. In: Computing, Control, Networking, Electronics and Embedded Systems Engineering (ICCNEEE), 2015 International Conference, pp 428–433

Google Scholar  

Alsaqaf K, Daneva M, Wieringa R (2017) Quality requirements in large-scale distributed agile projects - a systematic literature review. International Working Conference on Requirements Engineering: Foundation for Software Quality, pp 219–234

Alsaqaf W, Daneva M, Wieringa R (2019) Quality requirements challenges in the context of large-scale distributed agile: an empirical study. Inf Softw Technol 110:39–55

Article   Google Scholar  

Anwar S et al (2014) User-centered design practices in scrum development process: a distinctive advantage? 2014 IEEE 17th International Multi-Topic Conference (INMIC), pp 161–166

Ayalew T, Kidane T, Carlsson B (2013) Identification and evaluation of security activities in agile projects. In: Proceedings of the 18th Nordic Conference on Secure IT Systems, pp 139–153

Azham Z, Ghani I, Ithnin N (2011) Security backlog in Scrum security practices. Software Engineering (MySec), 2011 5th Malaysian Conference, pp 414–417

Behutiye W, Karhapää P, Costal D, Oivo M, Franch X (2017) Non-functional requirements documentation in agile software development: Challenges and solution proposal. International Conference on Product-Focused Software Process Improvement, pp 515–522

Behutiye W et al (2019) Management of quality requirements in agile and rapid software development: a systematic mapping study. Inf Softw Technol:106225

Bourimi M, Kesdogan D (2013) Experiences by using AFFINE for building collaborative applications for online communities. International Conference on Online Communities and Social Computing, pp 345–354

Bourimi M et al (2010) AFFINE for enforcing earlier consideration of NFRs and human factors when building socio technical systems following agile methodologies. In: International Conference on Human-Centered Software Engineering, pp 182–189

Bourque P, Fairley RE (2014) Guide to the software engineering body of knowledge (SWEBOK (R)): Version 3.0. IEEE Computer Society Press

Butt S, Ahmad WFW, Rahim L (2014) Handling tradeoffs between agile and usability methods. In International Conference on Computer and Information Science (ICCOINS), pp 1–6

Cajander Å, Larusdottir M, Gulliksen J (2013) Existing but not explicit - the user perspective in scrum projects in practice. In: IFIP Conference on Human-Computer Interaction, pp 762–779

Cannizzo F, Clutton R, Ramesh C (2008) Pushing the boundaries of testing and continuous integration. Agile Conference, AGILE'08, pp 501–505

Cao L, Ramesh B (2008) Agile requirements engineering practices: an empirical study. IEEE Softw 25(1):60–67

Chen L (2015) Continuous delivery: huge benefits, but challenges too. IEEE Softw 32(2):50–54

CMMI Product team (2010) CMMI for Development. Version 1.3. In: Software Eng Inst

Cruzes DS, Dybå T (2011) Recommended steps for thematic synthesis in software engineering. In: 2011 International Symposium on Empirical Software Engineering and Measurement:275–284

Cruzes DS, Felderer M, Oyetoyan TD, Gander M, Pekaric I (2017) How is security testing done in agile teams? A cross-case analysis of four software teams. In: Proceedings of the International Conference on Agile Software Development (XP), pp 201–216

Curcio K, Navarro T, Malucelli A, Reinher S (2018) Requirements engineering: a systematic mapping study in agile software development. J Syst Softw 139:32–50

Dybå T, Sjøberg D, Cruzes DS (2012) What works for whom, where, when, and why? On the role of context in empirical software engineering. In: ACM-IEEE International Symposium on Empirical Software Engineering and Measurement pp 19–28

Farid W (2012) The NORMAP methodology: lightweight engineering of non-functional requirements for agile processes. In: 2012 19th Asia-Pacific Software Engineering Conference, vol 1, pp 322–325

Chapter   Google Scholar  

Farid W, Mitropoulos F (2013) NORPLAN: non-functional requirements planning for agile processes. In Proceedings of SouthEastCon, pp 1–8

Fowler M, Highsmith J (2001) The agile manifesto. Softw Dev 9(8):28–35

Gary K, Enquobahrie A, Ibanez L, Cheng P, Yaniv Z, Cleary K et al (2011) Agile methods for open source safety-critical software. Softw: Pract Experience 41(9):945–962

Glass RL (1998) Defining quality intuitively. IEEE Softw 15(3):103–104

Heck P, Zaidman A (2016) A systematic literature review on quality criteria for agile requirements specifications. Softw Qual J:1–34

Heikkilä VT, Damian D, Lassenius C, Paasivaara M (2015) A mapping study on requirements engineering in agile software development. In: 41st Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp 199–207

Inayat I et al (2015) A systematic literature review on agile requirements engineering practices and challenges. Comput Hum Behav 51:915–929

Kalenda M, Hyna P, Rossi B (2018) Scaling agile in large organizations: practices, challenges, and success factors. J Softw-Evol Proc 30(10):e1954

Käpyaho M, Kauppinen M (2015) Agile requirements engineering with prototyping: a case study. In: 2015 IEEE 23rd International Requirements Engineering Conference (RE), pp 334–343

Karhapää P, Haghighatkhah A, Oivo M (2017) What do we know about alignment of requirements engineering and software testing? in: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, pp 354–363

Kasauli R, Liebel G, Knauss E, Gopakumar S, Kanagwa B (2017) Requirements engineering challenges in large-scale agile system development. 2017 IEEE 25th International Requirements Engineering Conference (RE), pp 352–361

Knauss E, Liebel G, Schneider K, Horkoff J, Kasauli R (2017) Quality requirements in agile as a knowledge management problem: more than just-in-time. In 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW), pp 427–430

Magües DA, Castor JW, Acuña ST (2016) Requirements engineering related usability techniques adopted in development process. In: SEKE 2016. Proceedings of the Twenty-Eight International Conference on Software Engineering and Knowledge Engineering. Knowledge Systems Institute Graduate School

Medeiros J, Vasconcelos A, Silva C, Goulão M (2020) Requirements specification for developers in agile projects: evaluation by two industrial case studies. Inf Softw Technol 117:106194

Merriam-Webster (2019) [Online]. Available at: https://www.merriam-webster.com/ . Accessed 17 April 2019

Miles MB, Huberman AM (1994) Qualitative data analysis: An expanded sourcebook, 2nd edn. Sage, Thousand Oaks, CA, p 1994

Mylopoulos J, Chung L, Nixon B (1992) Representing and using nonfunctional requirements: a process-oriented approach. IEEE Trans Softw Eng 18(6):483–497

Nawrocki M, Ochodek J, Jurkiewicz S, Alchimowicz B (2014) Agile requirements engineering: a research perspective. In: International Conference on Current Trends in Theory and Practice of Informatics, pp 40–51

Nvivo (2018) [Online]. Available at: http://www.qsrinternational.com/nvivo/nvivo-products . Accessed 27 Aug 2018

Petersen K, Wohlin C (2009) Context in industrial software engineering research. In: Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measurement, pp 401–404

Pohl K (2016) Requirements engineering fundamentals: a study guide for the certified professional for requirements engineering exam-foundation level-IREB compliant. Rocky Nook Inc

Rafi U, Mustafa T, Iqbal N, Zafar WUI (2015) US-Scrum: a methodology for developing software with enhanced correctness, usability and security. Int J Sci Eng Res 6(9):377

Ramesh B, Cao L, Baskerville R (2010) Agile requirements engineering practices and challenges: an empirical study. Inf Syst J 20(5):449–480

Rodríguez P, Yagüe A, Alarcón P, Garbajosa J (2009) Some findings concerning requirements in agile methodologies. In: Product-Focused Software Process Improvement, pp 171–184

Rodriguez P, Markkula J, Oivo M, Turula K (2012) Survey on agile and lean usage in Finnish software industry. In: Proceedings of the ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, pp 139–148

Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131

Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering: guidelines and examples. Wiley

Savolainen J, Kuusela J, Vilavaara A (2010) Transition to agile development-rediscovery of important requirements engineering practices. 2010 18th IEEE International Requirements Engineering Conference, pp 289–294

Schön E-M, Thomaschewski J, Escalona MJ (2017) Agile requirements engineering: a systematic literature review. Comput Stand Interfaces 49:79–91

Shaw M (1996) Truth vs knowledge: the difference between what a component does and what we know it does. In: 8th International Workshop on Software Specification and Design, p 181

Singh M (2008) U-SCRUM: an agile methodology for promoting usability. In: AGILE'08 Conference, pp 555–560

Siponen M, Baskerville R, Kuivalainen T (2005) Integrating security into agile development methods. In: Proceedings of the 38th Annual Hawaii International Conference on System Sciences, HICSS'05, p 185a

Sjøberg DIK, Dybå T, Anda BCD, Hannay JE (2008) In: Shull F et al (eds) Building theories in software engineering. Guide to Advanced Empirical Software Engineering. Springer

Stålhane T, Myklebust T, Hanssen GK (2012) The application of safe scrum to IEC 61508 certifiable software. In: 11th International Probabilistic Safety Assessment and Management Conference and the Annual European Safety and Reliability Conference, pp 6052–6061

Türpe S, Poller A (2017) Managing security work in Scrum: tensions and challenges. SecSE@ESORICS 2017:34–49

VersionOne, Inc (2016) 10th annual state of agile development survey. [online]. Available at: https://www.stateofagile.com/#ufh-i-338498988-10th-annual-stateof-agile-report/473508 . Accessed April 2019

Wagner S (2013) Software product quality control. Springer, Berlin

Book   Google Scholar  

Wale-Kolade A, Nielsen P, Päivärinta T (2014) Integrating usability practices into agile development: A case study. Proceedings of 23rd International Conference on Information Systems Development, pp 337–347

Wiegers KE, Beatty J (2013) Software requirements. Pearson Education

Download references

Acknowledgements

We would like to thank all the practitioners who participated in the interviews during this study. This work was supported by the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement 732253.

Open Access funding provided by University of Oulu including Oulu University Hospital.

Author information

Authors and affiliations.

M3S, Faculty of Information Technology and Electrical Engineering (ITEE), University of Oulu, 90014, Oulu, Finland

Pertti Karhapää, Woubshet Behutiye & Markku Oivo

Department of Languages, Computer Systems and Software Engineering, Faculty of Computer Science, Universidad Politécnica de Madrid, Madrid, Spain

Pilar Rodríguez

Universitat Politècnica de Catalunya, UPC-BarcelonaTech, 08034, Barcelona, Spain

Dolors Costal

Department of Service and Information System Engineering, UPC-BarcelonaTech, 08034, Barcelona, Spain

Xavier Franch

Nokia Oy, 90620, Oulu, Finland

Sanja Aaramaa

ITTI Sp. z o.o, 61-612, Poznan, Poland

Michał Choraś

University of Science and Technology, UTP Univerisy of Science and Technology, 5-796, Bydgoszcz, Poland

Bittium Wireless Ltd., 90590, Oulu, Finland

Jari Partanen

Softeam, 75016, Paris, France

Antonin Abherve

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Pertti Karhapää .

Additional information

Communicated by Jon Whittle

Appendix 1 Interview script for first round of interviews

Appendix 2 Interview script for second round of interviews

Appendix 3 Pre-questionnaire

Appendix 4. Beneficial practices for QR management in ASD

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Karhapää, P., Behutiye, W., Rodríguez, P. et al. Strategies to manage quality requirements in agile software development: a multiple case study. Empir Software Eng 26 , 28 (2021). https://doi.org/10.1007/s10664-020-09903-x

Download citation

Accepted : 11 September 2020

Published : 03 March 2021

DOI : https://doi.org/10.1007/s10664-020-09903-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Agile software development
• Requirements engineering
• Quality requirements
• Requirements management practices
• Non-functional requirements
• Agile requirements engineering
• Find a journal
• Publish with us
• Track your research

The Influence of Organizational Structure On Software Quality: An Empirical Case Study

• Nachiappan Nagappan ,
• Brendan Murphy ,
• Victor Basili ,
• Nachi Nagappan

MSR-TR-2008-11 | January 2008

Publication

Often software systems are developed by organizations consisting of many teams of individuals working together. Brooks states in the Mythical Man Month book that product quality is strongly affected by organization structure. Unfortunately there has been little empirical evidence to date to substantiate this assertion. In this paper we present a metric scheme to quantify organizational complexity, in relation to the product development process to identify if the metrics impact failure-proneness. In our case study, the organizational metrics when applied to data from Windows Vista were statistically significant predictors of failure-proneness. The precision and recall measures for identifying failure-prone binaries, using the organizational metrics, was significantly higher than using traditional metrics like churn, complexity, coverage, dependencies, and pre-release bug measures that have been used to date to predict failure-proneness. Our results provide empirical evidence that the organizational metrics are related to, and are effective predictors of failure-proneness.

• Follow on Twitter
• Like on Facebook
• Follow on LinkedIn
• Subscribe on Youtube
• Follow on Instagram
• Subscribe to our RSS feed

Share this page:

• Share on Twitter
• Share on Facebook
• Share on LinkedIn
• Share on Reddit

Sign up for the newsletter

Digital editions.

Total quality management: three case studies from around the world

With organisations to run and big orders to fill, it’s easy to see how some ceos inadvertently sacrifice quality for quantity. by integrating a system of total quality management it’s possible to have both.

Top 5 ways to manage the board during turbulent times Top 5 ways to create a family-friendly work culture Top 5 tips for a successful joint venture Top 5 ways managers can support ethnic minority workers Top 5 ways to encourage gender diversity in the workplace  Top 5 ways CEOs can create an ethical company culture Top 5 tips for going into business with your spouse Top 5 ways to promote a healthy workforce Top 5 ways to survive a recession Top 5 tips for avoiding the ‘conference vortex’ Top 5 ways to maximise new parents’ work-life balance with technology Top 5 ways to build psychological safety in the workplace Top 5 ways to prepare your workforce for the AI revolution Top 5 ways to tackle innovation stress in the workplace Top 5 tips for recruiting Millennials

There are few boardrooms in the world whose inhabitants don’t salivate at the thought of engaging in a little aggressive expansion. After all, there’s little room in a contemporary, fast-paced business environment for any firm whose leaders don’t subscribe to ambitions of bigger factories, healthier accounts and stronger turnarounds. Yet too often such tales of excess go hand-in-hand with complaints of a severe drop in quality.

Food and entertainment markets are riddled with cautionary tales, but service sectors such as health and education aren’t immune to the disappointing by-products of unsustainable growth either. As always, the first steps in avoiding a catastrophic forsaking of quality begins with good management.

There are plenty of methods and models geared at managing the quality of a particular company’s goods or services. Yet very few of those models take into consideration the widely held belief that any company is only as strong as its weakest link. With that in mind, management consultant William Deming developed an entirely new set of methods with which to address quality.

Deming, whose managerial work revolutionised the titanic Japanese manufacturing industry, perceived quality management to be more of a philosophy than anything else. Top-to-bottom improvement, he reckoned, required uninterrupted participation of all key employees and stakeholders. Thus, the total quality management (TQM) approach was born.

All in Similar to the Six Sigma improvement process, TQM ensures long-term success by enforcing all-encompassing internal guidelines and process standards to reduce errors. By way of serious, in-depth auditing – as well as some well-orchestrated soul-searching – TQM ensures firms meet stakeholder needs and expectations efficiently and effectively, without forsaking ethical values.

By opting to reframe the way employees think about the company’s goals and processes, TQM allows CEOs to make sure certain things are done right from day one. According to Teresa Whitacre, of international consulting firm ASQ , proper quality management also boosts a company’s profitability.

“Total quality management allows the company to look at their management system as a whole entity — not just an output of the quality department,” she says. “Total quality means the organisation looks at all inputs, human resources, engineering, production, service, distribution, sales, finance, all functions, and their impact on the quality of all products or services of the organisation. TQM can improve a company’s processes and bottom line.”

Embracing the entire process sees companies strive to improve in several core areas, including: customer focus, total employee involvement, process-centred thinking, systematic approaches, good communication and leadership and integrated systems. Yet Whitacre is quick to point out that companies stand to gain very little from TQM unless they’re willing to go all-in.

“Companies need to consider the inputs of each department and determine which inputs relate to its governance system. Then, the company needs to look at the same inputs and determine if those inputs are yielding the desired results,” she says. “For example, ISO 9001 requires management reviews occur at least annually. Aside from minimum standard requirements, the company is free to review what they feel is best for them. While implementing TQM, they can add to their management review the most critical metrics for their business, such as customer complaints, returns, cost of products, and more.”

The customer knows best: AtlantiCare TQM isn’t an easy management strategy to introduce into a business; in fact, many attempts tend to fall flat. More often than not, it’s because firms maintain natural barriers to full involvement. Middle managers, for example, tend to complain their authority is being challenged when boots on the ground are encouraged to speak up in the early stages of TQM. Yet in a culture of constant quality enhancement, the views of any given workforce are invaluable.

AtlantiCare in numbers

5,000 Employees

$280m Profits before quality improvement strategy was implemented

$650m Profits after quality improvement strategy

One firm that’s proven the merit of TQM is New Jersey-based healthcare provider AtlantiCare . Managing 5,000 employees at 25 locations, AtlantiCare is a serious business that’s boasted a respectable turnaround for nearly two decades. Yet in order to increase that margin further still, managers wanted to implement improvements across the board. Because patient satisfaction is the single-most important aspect of the healthcare industry, engaging in a renewed campaign of TQM proved a natural fit. The firm chose to adopt a ‘plan-do-check-act’ cycle, revealing gaps in staff communication – which subsequently meant longer patient waiting times and more complaints. To tackle this, managers explored a sideways method of internal communications. Instead of information trickling down from top-to-bottom, all of the company’s employees were given freedom to provide vital feedback at each and every level.

AtlantiCare decided to ensure all new employees understood this quality culture from the onset. At orientation, staff now receive a crash course in the company’s performance excellence framework – a management system that organises the firm’s processes into five key areas: quality, customer service, people and workplace, growth and financial performance. As employees rise through the ranks, this emphasis on improvement follows, so managers can operate within the company’s tight-loose-tight process management style.

After creating benchmark goals for employees to achieve at all levels – including better engagement at the point of delivery, increasing clinical communication and identifying and prioritising service opportunities – AtlantiCare was able to thrive. The number of repeat customers at the firm tripled, and its market share hit a six-year high. Profits unsurprisingly followed. The firm’s revenues shot up from $280m to $650m after implementing the quality improvement strategies, and the number of patients being serviced dwarfed state numbers.

Hitting the right notes: Santa Cruz Guitar Co For companies further removed from the long-term satisfaction of customers, it’s easier to let quality control slide. Yet there are plenty of ways in which growing manufacturers can pursue both quality and sales volumes simultaneously. Artisan instrument makers the Santa Cruz Guitar Co (SCGC) prove a salient example. Although the California-based company is still a small-scale manufacturing operation, SCGC has grown in recent years from a basement operation to a serious business.

SCGC in numbers

14 Craftsmen employed by SCGC

800 Custom guitars produced each year

Owner Dan Roberts now employs 14 expert craftsmen, who create over 800 custom guitars each year. In order to ensure the continued quality of his instruments, Roberts has created an environment that improves with each sale. To keep things efficient (as TQM must), the shop floor is divided into six workstations in which guitars are partially assembled and then moved to the next station. Each bench is manned by a senior craftsman, and no guitar leaves that builder’s station until he is 100 percent happy with its quality. This product quality is akin to a traditional assembly line; however, unlike a traditional, top-to-bottom factory, Roberts is intimately involved in all phases of instrument construction.

Utilising this doting method of quality management, it’s difficult to see how customers wouldn’t be satisfied with the artists’ work. Yet even if there were issues, Roberts and other senior management also spend much of their days personally answering web queries about the instruments. According to the managers, customers tend to be pleasantly surprised to find the company’s senior leaders are the ones answering their technical questions and concerns. While Roberts has no intentions of taking his manufacturing company to industrial heights, the quality of his instruments and high levels of customer satisfaction speak for themselves; the company currently boasts one lengthy backlog of orders.

A quality education: Ramaiah Institute of Management Studies Although it may appear easier to find success with TQM at a boutique-sized endeavour, the philosophy’s principles hold true in virtually every sector. Educational institutions, for example, have utilised quality management in much the same way – albeit to tackle decidedly different problems.

The global financial crisis hit higher education harder than many might have expected, and nowhere have the odds stacked higher than in India. The nation plays home to one of the world’s fastest-growing markets for business education. Yet over recent years, the relevance of business education in India has come into question. A report by one recruiter recently asserted just one in four Indian MBAs were adequately prepared for the business world.

RIMS in numbers

9% Increase in test scores post total quality management strategy

22% Increase in number of recruiters hiring from the school

20,000 Increase in the salary offered to graduates

50,000 Rise in placement revenue

At the Ramaiah Institute of Management Studies (RIMS) in Bangalore, recruiters and accreditation bodies specifically called into question the quality of students’ educations. Although the relatively small school has always struggled to compete with India’s renowned Xavier Labour Research Institute, the faculty finally began to notice clear hindrances in the success of graduates. The RIMS board decided it was time for a serious reassessment of quality management.

The school nominated Chief Academic Advisor Dr Krishnamurthy to head a volunteer team that would audit, analyse and implement process changes that would improve quality throughout (all in a particularly academic fashion). The team was tasked with looking at three key dimensions: assurance of learning, research and productivity, and quality of placements. Each member underwent extensive training to learn about action plans, quality auditing skills and continuous improvement tools – such as the ‘plan-do-study-act’ cycle.

Once faculty members were trained, the team’s first task was to identify the school’s key stakeholders, processes and their importance at the institute. Unsurprisingly, the most vital processes were identified as student intake, research, knowledge dissemination, outcomes evaluation and recruiter acceptance. From there, Krishnamurthy’s team used a fishbone diagram to help identify potential root causes of the issues plaguing these vital processes. To illustrate just how bad things were at the school, the team selected control groups and administered domain-based knowledge tests.

The deficits were disappointing. A RIMS students’ knowledge base was rated at just 36 percent, while students at Harvard rated 95 percent. Likewise, students’ critical thinking abilities rated nine percent, versus 93 percent at MIT. Worse yet, the mean salaries of graduating students averaged $36,000, versus $150,000 for students from Kellogg. Krishnamurthy’s team had their work cut out.

To tackle these issues, Krishnamurthy created an employability team, developed strategic architecture and designed pilot studies to improve the school’s curriculum and make it more competitive. In order to do so, he needed absolutely every employee and student on board – and there was some resistance at the onset. Yet the educator asserted it didn’t actually take long to convince the school’s stakeholders the changes were extremely beneficial.

“Once students started seeing the results, buy-in became complete and unconditional,” he says. Acceptance was also achieved by maintaining clearer levels of communication with stakeholders. The school actually started to provide shareholders with detailed plans and projections. Then, it proceeded with a variety of new methods, such as incorporating case studies into the curriculum, which increased general test scores by almost 10 percent. Administrators also introduced a mandate saying students must be certified in English by the British Council – increasing scores from 42 percent to 51 percent.

By improving those test scores, the perceived quality of RIMS skyrocketed. The number of top 100 businesses recruiting from the school shot up by 22 percent, while the average salary offers graduates were receiving increased by $20,000. Placement revenue rose by an impressive $50,000, and RIMS has since skyrocketed up domestic and international education tables.

No matter the business, total quality management can and will work. Yet this philosophical take on quality control will only impact firms that are in it for the long haul. Every employee must be in tune with the company’s ideologies and desires to improve, and customer satisfaction must reign supreme.

Contributors

• Industry Outlook

• Pharmaceutical Engineering Magazine
• Online Exclusives
• Special Reports
• Facilities & Equipment
• Information Systems
• Product Development
• Production Systems
• Quality Systems
• Regulatory Compliance
• Research + Development
• Supply Chain Management
• White Papers
• iSpeak Blog
• Editorial Calendar
• Article of the Year
• Submit an Article
• Editorial Team

Quality Risk Management for Biopharmaceuticals

In the dynamic and highly regulated world of biopharmaceutical manufacturing, maintaining and ensuring quality is a critical success factor. An effective quality risk management (QRM) system is a key component in the overall quality management infrastructure of biopharmaceutical organizations. It offers a structured, scientific, and risk-based approach to decision-making, addressing potential quality issues during manufacturing. High performing organizations effectively implement QRM into overall quality policies and procedures to enhance and streamline decision-making.

Implementing a robust QRM system is more than just a compliance requirement. It fundamentally contributes to the organization’s commitment to patient safety, product quality, and data integrity. A robust QRM system consists of key characteristics with clearly defined processes that contribute to the system’s success.

Reviewing the Risk Compliance Data

The following graphical data shows the relative compliance risk for pharmaceutical manufacturing organizations based on US Food and Drug Administration (FDA) regulatory activity (see Figure 1). Monitoring regulatory trends based on actual FDA activity provides useful insight for evaluating internal quality management system performance and proactively identifying areas of opportunity to improve overall compliance. Six major pharmaceutical regulation subparts are charted with relative annual activity increasing significantly from 2016 to 2020. During that period, the Building and Facilities, Laboratory Controls, and Production and Process Controls subparts were the largest areas, receiving 483 observations during regulatory inspections.

Those areas present increased compliance risk that would benefit from a formal review, gap analysis, and remediation to improve overall quality system performance and serve as priorities for time and resources. Regulatory risk during the COVID-19 pandemic decreased dramatically as the FDA performed few, if any, on-site investigations. However, activity during 2022 represents renewed on-site investigations with associated regulatory risk.

Figure 2 provides an annual trend of the top cited pharmaceutical regulations from 483 observations during regulatory investigations. These regulations are from the subparts identified in Figure 1 and the largest contributors to pharmaceutical regulatory risk. Any efforts to evaluate biopharmaceutical risk should consider the specific requirements identified in these regulations and address gaps identified during formal review and gap analysis as part of a QRM plan.

A Case Study

This case study concerns a major biopharmaceutical organization that specializes in producing monoclonal antibodies (mAbs) used in the treatment of various autoimmune diseases. As part of their commitment to quality and regulatory compliance, they have implemented a robust QRM system.

The organization’s production engineers identified a potential risk in their manufacturing process. The risk was related to variability in the cell culture phase, which could potentially lead to inconsistencies in the final product’s efficacy and safety.

Risk Identification

The QRM team initiated the risk identification process using failure mode and effects analysis (FMEA) and brainstorming sessions with cross-functional teams. They identified key risk factors, such as pH imbalances, temperature fluctuations, and contamination risks during the cell culture phase.

Risk Assessment

Using a risk matrix, the team assessed the potential impact and likelihood of each identified risk. They determined that temperature fluctuations posed the highest risk due to their high likelihood and potential to significantly impact product quality.

Risk Control

The organization decided to implement additional control measures to mitigate this risk:

• Enhanced monitoring: Installing advanced temperature monitoring systems with automatic alerts for deviations.
• Process improvement: Optimizing the cell culture process to be more robust against minor temperature changes.
• Employee training: Conducting extensive training for staff on the importance of maintaining optimal temperature conditions.

Risk Communication

The QRM team communicated the identified risks, their potential impact, and the planned control measures to all relevant stakeholders, including the manufacturing team, quality assurance department, and senior management.

Implementation

The proposed measures were implemented, and their effectiveness was closely monitored. This included regular review meetings and updates to the risk management plan.

The new control measures led to a significant reduction in temperature-related variability in the cell culture process. As a result, the consistency and quality of the mAbs improved, leading to enhanced patient safety and regulatory compliance.

Lessons Learned

The proactive approach to identifying and managing a critical risk in their manufacturing process demonstrated the importance of a dynamic and integrated QRM system. The case also highlighted the need for continuous monitoring and improvement in risk management practices.

Case Study Conclusion

This case study exemplifies the application of a structured QRM process in the biopharmaceutical industry. It illustrates the importance of identifying, assessing, controlling, and communicating risks in a systematic manner to ensure the production of high-quality biopharmaceutical products.

Characteristics of a Biopharmaceutical Qrm System

Identification of risk is a cross-functional effort that begins in the late development stages prior to technology transfer. In the early stages, research and development (R&D) is the main contributor in the risk identification process, which is facilitated by quality and manufacturing who are participants. As manufacturing develops detailed knowledge of the new process and technology, it provides a strong perspective on potential issues and risks that may exist in day-to-day manufacturing. At this time, all teams must compromise to ensure the final technology and process transfer meet the strategic goals of launching a new product.

Once the technology transfer is complete, manufacturing takes the lead in monitoring risk, along with quality. The manufacturing team also proposes any potential changes, which are reviewed by R&D, quality, and, possibly, commercial participants. Performance metrics are developed jointly between quality and manufacturing and used to periodically report to cross-functional leaders.

The main characteristics of a robust QRM system for biopharmaceutical manufacturers are identified in the following sections.

The initial step in any QRM system is the identification of potential risks. It is necessary to understand what could potentially go wrong in the manufacturing process to manage and mitigate these risks effectively. Elevated performance in risk identification is demonstrated by organizations conducting risk identification with input from cross-functional subject matter experts.

This typically involves brainstorming sessions with relevant stakeholders, analysis of historical data and problem reports, and reviews of process documentation. Clear guidelines should be established for what constitutes a risk, and all identified risks should be documented and maintained in a risk register.

In addition to brainstorming sessions and historical data analysis, other tools such as FMEA, hazard identification, or process hazard analysis can be implemented for a systematic approach. Expert opinions and predictive models can also be used. A successful process should also involve reassessing the risk landscape periodically and after any significant changes. Changes requiring revalidation are a notable trigger to update risk profiles.

A robust biopharmaceutical QRM system recognizes that the process of risk identification is continuous and dynamic, adjusting to changes in procedures, equipment, materials, and the overall business environment. It also considers both internal and external sources of risk.

After identifying risks, it is crucial to evaluate them in terms of their potential impact on product quality and the probability of their occurrence. This allows the company to prioritize its risk management efforts.

Risk assessment usually involves qualitative or quantitative methods. Qualitative methods might include rating risks on a scale from low to high, whereas quantitative methods might involve statistical analysis or simulation. Risk assessment is about creating an informed understanding of the risk and considering the severity of the impact, the likelihood of occurrence, and the detectability of the risk. This aids in prioritizing resources and efforts for risk control.

The process should include risk ranking or scoring systems that can objectively evaluate and compare different risks. Detailed risk maps or matrices can be created to visualize the risk landscape. Risk assessments should be periodically reviewed and updated, especially when new information becomes available.

This step involves deciding on and implementing measures to mitigate the identified risks. Without this step, the risk management process would be incomplete. Risk control involves not only mitigating risks but also deciding whether to accept, transfer, or avoid certain risks. Risk control measures should be proportional to the significance of the risk.

Risk control could involve anything from making changes to the manufacturing process to training employees in new procedures. A key part of this step is documenting the control measures and monitoring their effectiveness over time. After devising risk control measures, a pilot test can be conducted for complex or high-stake measures to ensure their effectiveness before full-scale implementation. The measures should also be reviewed and updated regularly, and particularly after any significant incidents.

Communication and Consultation

Effective communication ensures all relevant stakeholders are aware of the risks and the steps being taken to control them. This not only fosters a culture of risk awareness, but also ensures risk management efforts are coordinated across the organization. Effective communication promotes a shared understanding of risks, risk management practices, and individual roles and responsibilities in managing risk. It should involve all levels of the organization, as well as external stakeholders when appropriate.

This could involve regular meetings, reports, or automated notifications. The key is to ensure that the right information reaches the right people at the right time. The communication process should be a two-way street, allowing feedback from all stakeholders. In addition to meetings and reports, knowledge management systems or collaboration platforms could be used to facilitate communication. Clear protocols should be established for escalation of high-priority risks.

Continuous Monitoring and Review

The risk landscape can change over time, with new risks emerging and old ones disappearing or changing in severity. Continuous monitoring and review ensure that a QRM system stays relevant and effective. Incorporating accurate trend data based on regulatory activity provides an additional level of input elevating the effectiveness of risk management activities.

This can involve regular risk assessments, audits, and reviews of risk control measures. Any changes should be documented and communicated to relevant stakeholders. Monitoring and review processes should include the risks themselves and the effectiveness of the QRM system, changes in context, and the identification of emerging risks.

Risk Management Integration

Risk management should be an integral part of all organizational processes—not a separate activity. This ensures risk considerations are a part of all decisions, rather than being an afterthought. Integrating risk management with other business processes ensures risk management is proactive rather than reactive. It allows risks to be addressed before they can cause problems.

This could involve incorporating risk management into existing process documentation, training employees on risk management, or establishing a risk management committee. This could involve the use of integrated management systems or embedding risk management into standard operating procedures. Cross-functional teams or committees could be established to oversee the integration. Key performance indicators related to risk management should be established and monitored. Audits and reviews should be scheduled regularly and triggered by significant changes or incidents. Feedback from these activities should be used to drive continuous improvement.

Root Cause Analysis

Understanding the root cause of a problem allows for more effective risk management. It helps avoid merely treating the symptoms of a problem, which can lead to recurrence. The goal of root cause analysis is to prevent recurrence of problems by addressing their underlying causes, not just the symptoms. It allows for more efficient use of resources and improves process understanding.

Techniques such as the five whys and fishbone diagrams, among others, can be used to identify root causes. Once identified, these root causes should be addressed in the risk control measures. When conducting root cause analysis, it is important to ensure a blame-free environment where all ideas are considered. Tools such as Pareto charts could be used to prioritize root causes. Root cause prioritization may also reference regulatory trends based on current regulatory activity. Corrective and preventive actions should be devised to address the root causes.

Data-Driven Decision-Making

Decisions about risk management should be based on data, not on gut feelings or intuition. This leads to more objective and effective decisions. The use of data promotes objectivity, consistency, and efficiency in decision-making. It also allows for tracking and demonstrating the performance of the QRM system. An example of data-driven decision-making used by high-performing organizations uses available newsletters, visualizations, and trend tracking regulatory data to provide accurate insights to compliance risk.

This might involve collecting and analyzing data on process performance, product quality, and the effectiveness of risk control measures. Decision-making tools such as decision trees or Bayesian networks can also be used. An effective process should include not only collection and analysis of data, but also data management practices to ensure data integrity and usability. Advanced data analytics or artificial intelligence could be used for predictive risk modeling. Regulatory trends and current regulatory activity are also indicators providing insight into predictive risks of regulatory audits.

Quality Culture

A strong culture of quality fosters individual accountability, intrinsic motivation, and proactive behavior in managing risk and it ensures risk management is not the responsibility of just the quality department. Successful organizations building a strong culture of quality and compliance have notable focus and support from executive leadership. A successful quality culture can only succeed with outstanding support from organizational executives. The “tone at the top” significantly drives the performance and adherence of the organization to quality principles.

This could involve training, recognition programs, or changes to organizational structure. It is important to regularly assess the culture of quality and adjust as needed. Activities to foster a quality culture could include workshops, training sessions, recognition programs, and team-building activities. Regular culture assessments could be conducted through surveys or interviews and the findings used to inform culture improvement initiatives.

A robust biopharmaceutical QRM system recognizes that the process of risk identification is continuous and dynamic, adjusting to changes in procedures, equipment, materials, and the overall business environment.

Flexibility and Adaptability

As the organization and its external environment change, the QRM system needs to be able to adapt. A rigid system that cannot handle change will quickly become ineffective. A flexible and adaptable QRM system allows the organization to respond effectively to changes and challenges, turning them into opportunities rather than threats. It helps ensure the system’s resilience and long-term sustainability. Using data-driven metrics and tracking tools facilitates effective management of quality and compliance risk.

This can involve regular reviews of the QRM system and a process for making changes to it. Feedback from stakeholders should be actively sought and incorporated. Scenario analysis or stress testing could be used to evaluate and improve the system’s adaptability. A change management process should be established to handle changes in a systematic and controlled manner. Integral to a change management process should be the incorporation of risk assessment and evaluation relevant to any proposed changes.

Compliance with Regulations

Biopharmaceutical companies operate in a heavily regulated environment. Compliance with regulations avoids legal problems and ensures products are safe and effective. It also promotes trust and credibility among stakeholders, and it provides a baseline for risk management practices.

Compliance can be ensured by keeping up to date with regulatory changes, incorporating these changes into the QRM system, and regularly auditing for compliance. Regular training should be provided to keep staff current on regulatory requirements. Regulatory intelligence activities could be conducted to anticipate and prepare for upcoming changes.

Compliance checks should be integrated into the risk assessment and review processes. Additionally, compliance reports and newsletters summarizing regulatory activity provide valuable insight into risks and trends associated with regulatory compliance for life sciences. Further expanding compliance data to broader time horizons increases insights into longer-term trends and the value of current trends in a historical perspective.

Traceability and Documentation

Documentation provides evidence of the QRM system’s functioning. It also allows for traceability, which is crucial for root cause analysis and for demonstrating compliance with regulations. Proper documentation allows the team to preserve institutional knowledge, learn from past experiences, and demonstrate compliance. Traceability is crucial for investigating incidents, validating processes, and ensuring product quality.

Documentation should be maintained for all risk management activities, including risk identification, assessment, and control. It should be kept in a format that is easily accessible and understandable. Traceability can be maintained through unique identifiers for risks and control measures, and by linking related documents. A document management system could be used to manage and control documents. The system should support version control, approval processes, and easy retrieval of documents. Traceability could be maintained through traceability matrices or dedicated software systems.

An effective QRM system in biopharmaceutical manufacturing is multifaceted, involving the identification and assessment of potential risks, robust control mechanisms, effective communication strategies, and regular monitoring and review procedures. The QRM system should be flexible and adaptable, grounded in data-driven decision-making, and deeply integrated within the organization’s culture and processes. The risk management process, which includes risk identification and mitigation, is a cross-functional effort requiring participation from R&D, quality, and manufacturing, with metrics for monitoring and reporting process effectiveness to cross-functional leaders.

Organizations performing at elevated levels consistently demonstrate an ability to incorporate risk criteria into daily operations using various tools to evaluate risk according to product and patient impact. Compliance with regulations and maintaining detailed traceability and documentation are also of paramount importance. Although implementing such a comprehensive system can be complex, the benefits of ensuring product quality and safety, and ultimately patient health, are profound. The successful deployment of QRM necessitates a continual commitment to each of these characteristics, fostering a culture of quality that permeates every aspect of the organization.

Not a Member Yet?

To continue reading this article and to take advantage of full access to Pharmaceutical Engineering magazine articles, technical reports, white papers and exclusive content on the latest pharmaceutical engineering news, join ISPE today. In addition to exclusive access to all of the content in Pharmaceutical Engineering magazine, you will get online access to 24 ISPE Good Practice Guides, exclusive networking events, regulatory resources, Communities of Practice, and more.

Learn more about the valuable benefits you'll receive with an ISPE membership.

Join Today Member Login

Related Articles

Advanced therapy medicinal products (ATMPs) and cell and gene therapies (C&GTs) represent a promising medical product class that employs gene therapy, cell therapy, or tissue engineering to address various diseases and injuries. One critical aspect of ATMP and C&GT manufacturing is using cell culture media. With thousands of ATMPs and C&GTs in clinical trial phases, the role of...

Antibody engineering has transformed the development of therapeutic antibodies, enabling the creation of specific and effective treatments for a range of diseases. These antibody-based therapeutics are advancing in clinical development at a rapid rate and are being approved in record numbers. Currently, more than 100 monoclonal antibodies (mAbs) have been approved for the treatment of various...

Although traditional tank farm systems have long been the cornerstone of buffer preparation, they face challenges that have grown with the expansion of processing scale in the industry.

• Open access
• Published: 09 May 2024

Examining the feasibility of assisted index case testing for HIV case-finding: a qualitative analysis of barriers and facilitators to implementation in Malawi

• Caroline J. Meek 1 , 2 ,
• Tiwonge E. Mbeya Munkhondya 3 ,
• Mtisunge Mphande 4 ,
• Tapiwa A. Tembo 4 ,
• Mike Chitani 4 ,
• Milenka Jean-Baptiste 2 ,
• Dhrutika Vansia 4 ,
• Caroline Kumbuyo 4 ,
• Jiayu Wang 2 ,
• Katherine R. Simon 4 ,
• Sarah E. Rutstein 5 ,
• Clare Barrington 2 ,
• Maria H. Kim 4 ,
• Vivian F. Go 2 &
• Nora E. Rosenberg 2  

BMC Health Services Research volume  24 , Article number:  606 ( 2024 ) Cite this article

83 Accesses

Metrics details

Assisted index case testing (ICT), in which health care workers take an active role in referring at-risk contacts of people living with HIV for HIV testing services, has been widely recognized as an evidence-based intervention with high potential to increase status awareness in people living with HIV. While the available evidence from eastern and southern Africa suggests that assisted ICT can be an effective, efficient, cost-effective, acceptable, and low-risk strategy to implement in the region, it reveals that feasibility barriers to implementation exist. This study aims to inform the design of implementation strategies to mitigate these feasibility barriers by examining “assisting” health care workers’ experiences of how barriers manifest throughout the assisted ICT process, as well as their perceptions of potential opportunities to facilitate feasibility.

In-depth interviews were conducted with 26 lay health care workers delivering assisted ICT in Malawian health facilities. Interviews explored health care workers’ experiences counseling index clients and tracing these clients’ contacts, aiming to inform development of a blended learning implementation package. Transcripts were inductively analyzed using Dedoose coding software to identify and describe key factors influencing feasibility of assisted ICT. Analysis included multiple rounds of coding and iteration with the data collection team.

Participants reported a variety of barriers to feasibility of assisted index case testing implementation, including sensitivities around discussing ICT with clients, privacy concerns, limited time for assisted index case testing amid high workloads, poor quality contact information, and logistical obstacles to tracing. Participants also reported several health care worker characteristics that facilitate feasibility (knowledge, interpersonal skills, non-stigmatizing attitudes and behaviors, and a sense of purpose), as well as identified process improvements with the potential to mitigate barriers.

Conclusions

Maximizing assisted ICT’s potential to increase status awareness in people living with HIV requires equipping health care workers with effective training and support to address and overcome the many feasibility barriers that they face in implementation. Findings demonstrate the need for, as well as inform the development of, implementation strategies to mitigate barriers and promote facilitators to feasibility of assisted ICT.

Trial registration

NCT05343390. Date of registration: April 25, 2022.

Peer Review reports

Introduction

To streamline progress towards its goal of ending AIDS as a public health threat by 2030, the Joint United Nations Programme on HIV/AIDS (UNAIDS) launched a set of HIV testing and treatment targets [ 1 ]. Adopted by United Nations member states in June 2021, the targets call for 95% of all people living with HIV (PLHIV) to know their HIV status, 95% of all PLHIV to be accessing sustained antiretroviral therapy (ART), and 95% of all people receiving ART to achieve viral suppression by 2025 [ 2 ]. Eastern and southern Africa has seen promising regional progress towards these targets in recent years, and the region is approaching the first target related to status awareness in PLHIV- in 2022, 92% of PLHIV in the region were aware of their status [ 3 ]. However, several countries in the region lag behind [ 4 ], and as 2025 approaches, it is critical to scale up adoption of evidence-based interventions to sustain and accelerate progress.

Index case testing (ICT), which targets provision of HIV testing services (HTS) for sexual partners, biological children, and other contacts of known PLHIV (“index clients”), is a widely recognized evidence-based intervention used to identify PLHIV by streamlining testing efforts to populations most at risk [ 5 , 6 , 7 ]. Traditional approaches to ICT rely on passive referral, in which index clients invite their contacts for testing [ 5 ]. However, the World Health Organization (WHO) and the President’s Emergency Plan for HIV/AIDS Relief (PEPFAR) have both recommended assisted approaches to ICT [ 6 , 8 , 9 , 10 ], in which health care workers (HCWs) take an active role in referral of at-risk contacts for testing, due to evidence of improved effectiveness in identifying PLHIV compared to passive approaches [ 10 , 11 , 12 , 13 , 14 ]. As a result, there have been several efforts to scale assisted ICT throughout eastern and southern Africa in recent years [ 15 , 16 , 17 , 18 , 19 , 20 ]. In addition to evidence indicating that assisted ICT can be effective in increasing HIV testing and case-finding [ 16 , 17 , 21 , 22 , 23 , 24 ], implementation evidence [ 25 ] from the region suggests that assisted ICT can be an efficient [ 14 ], acceptable [ 5 , 13 , 15 , 18 , 20 , 21 , 26 ], cost-effective [ 27 ], and low-risk [ 21 , 22 , 24 , 28 , 29 ] strategy to promote PLHIV status awareness. However, the few studies that focus on feasibility, or the extent to which HCWs can successfully carry out assisted ICT [ 25 ], suggest that barriers exist to feasibility of effective implementation [ 18 , 19 , 20 , 30 , 31 , 32 ]. Developing informed implementation strategies to mitigate these barriers requires more detailed examination of how these barriers manifest throughout the assisted ICT process, as well as of potential opportunities to facilitate feasibility, from the perspective of the HCWs who are doing the “assisting”.

This qualitative analysis addresses this need for further detail by exploring “assisting” HCWs’ perspectives of factors that influence the feasibility of assisted ICT, with a unique focus on informing development of effective implementation strategies to best support assisted ICT delivery in the context of an implementation science trial in Malawi.

This study was conducted in the Machinga and Balaka districts of Malawi. Malawi is a country in southeastern Africa in which 7.1% of the population lives with HIV and 94% of PLHIV know their status [ 4 ]. Machinga and Balaka are two relatively densely populated districts in the southern region of Malawi [ 33 ] with HIV prevalence rates similar to the national average [ 34 ]. We selected Machinga and Balaka because they are prototypical of districts in Malawi implementing Ministry of Health programs with support from an implementing partner.

Malawi has a long-established passive ICT program, and in 2019 the country also adopted an assisted component, known as voluntary assisted partner notification, as part of its national HIV testing policy [ 32 ]. In Malawi, ICT is conducted through the following four methods, voluntarily selected by the index client: 1) passive referral, in which HCWs encourage the index client to refer partners for voluntary HTS, 2) contract referral, in which HCWs establish an informal ‘contract’ with index clients that agrees upon a date that the HCW can contact the contact clients if they have not yet presented for HTS; 3) provider referral, in which HCWs contact and offer voluntary HTS to contact clients; and 3) dual referral, in which HCWs accompany and provide support to index clients in disclosing their status and offering HTS to their partners [ 8 ]. 

While Malawi has one of the lowest rates of qualified clinical HCWs globally (< 5 clinicians per 100,000 people) [ 35 ], the country has a strong track record of shifting HTS tasks to lay HCWs, who have been informally trained to perform certain health care delivery functions but do not have a formal professional/para-professional certification or tertiary education degree, in order to mitigate this limited medical workforce capacity [ 32 , 36 ]. In Malawi, lay HCW roles include HIV Diagnostic Assistants (who are primarily responsible for HIV testing and counseling, including index case counseling) and community health workers (who are responsible for a wider variety of tasks, including index case counseling and contact tracing) [ 32 ]. Non-governmental organization implementing partners, such as the Tingathe Program, play a critical role in harnessing Malawian lay HCW capacity to rapidly and efficiently scale up HTS, including assisted ICT [ 32 , 37 , 38 , 39 ].

Study design

Data for this analysis were collected as part of formative research for a two-arm cluster randomized control trial examining a blended learning implementation package as a strategy for building HCW capacity in assisted ICT [ 40 ]. Earlier work [ 32 ] established the theoretical basis for testing the blended learning implementation package, which combines individual asynchronous modules with synchronous small-group interactive sessions to enhance training and foster continuous quality improvement. The formative research presented in this paper aimed to further explore factors influencing feasibility of the assisted ICT from the perspective of HCWs in order to inform development of the blended learning implementation package.

Prior to the start of the trial (October-December 2021), the research team conducted 26 in-depth interviews (IDIs) with lay HCWs at 14 of the 34 facilities included in the parent trial. We purposively selected different types of facilities (hospitals, health centers, and dispensaries) in both districts and from both randomization arms, as this served as a qualitative baseline for a randomized trial. Within these facilities, we worked with facility supervisors to purposively select HCWs who were actively engaged in Malawi’s ICT program from the larger sample of HCWs eligible for the parent trial (had to be at least 18 years old, employed full-time at one of the health facilities included in the parent trial, and involved in counseling index clients and/or tracing their contacts). The parent trial enrolled 306 HCWs, who were primarily staff hired by Tingathe Program to support facilities implementing Malawi’s national HIV program.

Data collection

IDIs were conducted by three trained Malawian interviewers in a private setting using a semi-structured guide. IDIs were conducted over the phone when possible ( n  = 18) or in-person at sites with limited phone service ( n  = 8). The semi-structured guide was developed for this study through a series of rigorous, iterative discussions among the research team (Additional file 1 ). The questions used for this analysis were a subset of a larger interview. The interview guide questions for this analysis explored HCWs’ experiences with assisted ICT, including barriers and facilitators to implementation. Probing separately about the processes of counseling index clients and tracing their contacts, interviewers asked questions such as “What is the first thing that comes to mind when you think of counseling index clients/tracing contacts?”, “What aspects do you [like/not like] about…?” and “What do your colleagues say about…?”. When appropriate, interviewers probed further about how specific factors mentioned by the participant facilitate or impede the ICT implementation experience.

The IDIs lasted from 60–90 min and were conducted in Chichewa, a local language in Malawi. Eleven audio recordings were transcribed verbatim in Chichewa before being translated into English and 15 recordings were directly translated and transcribed into English. Interviewers summarized each IDI after it was completed, and these summaries were discussed with the research team routinely.

Data analysis

The research team first reviewed all of the interview summaries individually and then met multiple times to discuss initial observations, refining the research question and scope of analysis. A US-based analyst (CJM) with training in qualitative analysis used an inductive approach to develop a codebook, deriving broad codes from the implementation factors mentioned by participants throughout their interviews. Along with focused examination of the transcripts, she consulted team members who had conducted the IDIs with questions or clarifications. CJM regularly met with Malawian team members (TEMM, MM, TAT) who possess the contextual expertise necessary to verify and enhance meaning. She used the Dedoose (2019) web application to engage in multiple rounds of coding, starting with codes representing broad implementation factors and then further refining the codebook as needed to capture the nuanced manifestations of these barriers and facilitators. Throughout codebook development and refinement, the analyst engaged in memoing to track first impressions, thought processes, and coding decisions. The analyst presented the codebook and multiple rounds of draft results to the research team. All transcripts and applied codes were also reviewed in detail by additional team members (MJB, DV). Additional refinements to the codebook and results interpretations were iteratively made based on team feedback.

Ethical clearance

Ethical clearance was provided by UNC’s IRB, Malawi’s National Health Sciences Research Committee, and the Baylor College of Medicine IRB. Written informed consent was obtained from all participants in the main study and interviewers confirmed verbal consent before starting the IDIs.

Participant characteristics are described in Table  1 below.

Factors influencing feasibility of assisted ICT: barriers and facilitators

Participants described a variety of barriers and facilitators to feasibility of assisted ICT, manifesting across the index client counseling and contact client tracing phases of the implementation process. Identified barriers included sensitivities around discussing ICT with clients, privacy concerns, limited time for ICT amid high workloads, poor quality contact information, and logistical obstacles to tracing. In addition to these barriers, participants also described several HCW characteristics that facilitated feasibility: ICT knowledge, interpersonal skills, positive attitudes towards clients, and sense of purpose. Barriers and facilitators are mapped to the ICT process in Fig.  1 and described in greater detail in further sections.

Conceptual diagram mapping feasibility barriers and facilitators to the ICT process

Feasibility barriers

Sensitivities around discussing ict with clients.

Participants described ICT as a highly sensitive topic to approach with clients. Many expressed a feeling of uncertainty around how open index clients will be to sharing information about their contacts, as well as how contacts will react when approached for HTS. When asked about difficult aspects of counseling index clients, many HCWs mentioned clients’ hesitance or declination to participate in assisted ICT and share their contacts. Further, several HCWs mentioned that some index clients would provide false contact information. These index client behaviors were often attributed to confidentiality concerns, fear of unwanted status disclosure, and fear of the resulting implications of status disclosure: “They behave that way because they think you will be telling other people about their status…they also think that since you know it means their life is done, you will be looking at them differently .” Populations commonly identified as particularly likely to hesitate, refuse, or provide false information included youth (described as “ shy ” “ thinking they know a lot ” and “ difficult to reveal their contacts ”) and newly diagnosed clients (“it may be hard for them to accept [their HIV diagnosis]” ). One participant suggested that efforts to pair index clients with same-sex HCWs could make them more comfortable to discuss their contacts.

When asked about the first things that come to mind when starting to trace contacts, many participants discussed wondering how they will be received by the contact and preparing themselves to approach the contact. When conducting provider or contract referral, HCWs described a variety of challenging reactions that can occur when they approach a contact for HTS- including delay or refusal of testing, excessive questioning about the identity of the index client who referred them for testing, and even anger or aggression. Particularly mentioned in the context of male clients, these kinds of reactions can lead to stress and uncertain next steps for HCWs: “I was very tensed up. I was wondering to myself what was going to happen…he was talking with anger.”

Participants also noted the unique sensitivities inherent in conducting dual referral and interacting with sexual partners of index clients, explaining that HIV disclosure can create acute conflict in couples due to perceived blame and assumptions of infidelity. They recounted these scenarios as particularly difficult to navigate, with high stakes that require high-quality counseling skills: “sometimes if you do not have good counseling the marriage happens to get to an end.” . Some participants discussed concern about index client risk of intimate partner violence (IPV) upon partner disclosure: “they think that if they go home and [disclose their HIV status], the marriage will end right there, or for some getting to a point of [being] beaten.”

Privacy concerns

Participants also reported that clients highly value privacy, which can be difficult to secure throughout the ICT process. In the facility, while participants largely indicated that counseling index clients was much more successful when conducted in a private area, many reported limited availability of private counseling space. One participant described this challenge: “ if I’m counseling an index client and people keep coming into the room…this compromises the whole thing because the client becomes uncomfortable in the end.” Some HCWs mentioned working around this issue through use of screens, “do-not-disturb” signs, outdoor spots, and tents.

Participants also noted maintaining privacy as a challenge when tracing contact clients in the field, as they sometimes find clients in a situation that is not conducive to private conversations. One participant described: “ we get to the house and find that there are 4, 5 people with our [contact client]…it doesn’t go well…That is a mission gone wrong. ” Participants also noted that HCWs are also often easily recognizable in the community due to their bikes and cars, which exacerbates the risk of compromising privacy. To address privacy challenges in the community, participants reported strategies to increase discretion, including dressing to blend in with the community, preparing an alternate reason to be looking for the client, and offering HTS to multiple people or households to avoid singling out one person.

Limited time for ICT amid high workloads

Some participants indicated that strained staffing capacity leads HCWs to have to perform multiple roles, expressing challenges in balancing their ICT work with their other tasks. As one participant described, “Sometimes it is found that you are assigned a task here at the hospital to screen anyone who comes for blood testing, but you are also supposed to follow up [with] the contacts the same day- so it becomes a problem…you fail to follow up [with] the contacts.” Some also described being the only, or one of few staff responsible for ICT: “You’re doing this work alone, so you can see that it is a big task to do it single-handedly.” The need to counsel each index client individually, as a result of confidentiality concerns, further increases workload for the limited staff assigned to this work. Further, HCWs often described contact tracing in the field as time-consuming and physically taxing, which leaves them less time and energy for counseling. Many HCWs noted the need to hire more staff dedicated to ICT work.

High workloads also resulted in shorter appointments and less time to counsel index clients, which participants reported limits the opportunity for rapport that facilitates openness or probes for detailed information about sexual partners. Participants emphasized the importance of having enough time to meaningfully engage with index clients: “For counseling you cannot have a limit to say, ‘I will talk to him for 5 min only.’ …That is not counseling then. You are supposed to stay up until…you feel that this [person] is fulfilled.” . In addition, high workload can reduce the capacity of HCWs to deliver quality counseling: “So you find that as you go along with the counseling, you can do better with the first three clients but the rest, you are tired and you do short cuts.”

High workloads also lead to longer queues, which may deter clients from coming into the clinic or cause them to leave before receiving services: “Sometimes because of shortage of staff, it happens that you have been assigned a certain task that you were supposed to do but at the same time there are clients who were supposed to be counseled. As a result, because you spent more time on the other task as a result you lose out some of the clients because you find that they have gone.” In response to long queues, several participants described ‘fast-tracking’ contact clients who come in for HTS in effort to maximize case-finding by prioritizing those who have been identified as at risk of HIV.

Poor quality contact information

Participants repeatedly discussed the importance of eliciting accurate information about a person’s sexual partners, including where, when, and how to best contact them. As one participant said, “ Once the index has given us the wrong information then everything cannot work, it becomes wrong…if he gives us full information [with] the right details then everything becomes successful and happens without a problem. ” Adequate information is a critical component of the ICT process, and incorrect or incomplete information delays or prevents communication with contact clients.

Inadequate information, which can include incorrect or incomplete names, phone numbers, physical addresses, and contextual details, can arise from a variety of scenarios. Most participants mentioned index clients providing incorrect information as a concern. This occurred either intentionally to avoid disclosure or unintentionally if information was not known. Poor quality contact information also results from insufficient probing and poor documentation, which is often exacerbated by aforementioned HCW time and energy constraints. In one participant’s words, “The person who has enlisted the contact…is the key person who can make sure that our tracing is made easy.” Participants noted the pivotal role of the original HCW who first interacts with the index client in not only eliciting correct locator information but also eliciting detailed contextual information. For example, details about a contact client’s profession are helpful to trace the client at a time when they will likely be at home. Other helpful information included nicknames, HIV testing history, and notes about confidentiality concerns.

Logistical obstacles to tracing

Some contact clients are reached by phone whereas others must be physically traced in the community. Some participants reported difficulty with tracing via phone, frequently citing network problems and lack of sufficient airtime allocated by the facility. Participants also reported that some clients were unreachable by phone, necessitating physical tracing. Physically tracing a contact client requires a larger investment of resources than phone tracing, especially when the client lives at a far distance from the clinic. Participants frequently discussed having to travel far distances to reach contact clients, an issue some saw as exacerbated by people who travel to clinics at far distances due to privacy concerns.

While most participants reported walking or biking to reach contact clients in the community, some mentioned using a motorcycle or Tingathe vehicle. However, access to vehicles is often limited and these transportation methods require additional expenses for fuel. Walking or biking was also reported to expose HCWs to inclement weather, including hot or rainy seasons, and potential safety risks such as violence.

Participants reported that traveling far distances can be physically taxing and time-consuming, sometimes rendering them too tired or busy to attend to other tasks. Frequent travel influenced HCW morale, particularly when a tracing effort did not result in successfully recruiting a contact client. Participants frequently described this perception of wasted time and energy as “ painful ”, with the level of distress often portrayed as increasing with the distance travelled. As one HCW said, “You [can] find out that he gave a false address. That is painful because it means you have done nothing for the person, you travelled for nothing.”

HCWs described multiple approaches used to strategically allocate limited resources for long distances. These approaches included waiting to physically trace until there are multiple clients in a particular area, reserving vehicle use for longer trips, and coordinating across HCWs to map out contact client locations. HCWs also mentioned provision of rain gear and sun protection to mitigate uncomfortable travel. Another approach involved allocating contact tracing to HCWs based in the same communities as the contact clients.

Feasibility facilitators

Hcw knowledge about ict.

Participants reported that HCWs with a thorough understanding of ICT’s rationale and purpose can facilitate client openness. Clients were more likely to engage with HCWs about assisted ICT if they understood the benefits to themselves and their loved ones. One HCW stated, “If the person understands why we need the information, they will give us accurate information.”

Participants also discussed the value of deep HCW familiarity with ICT procedures and processes, particularly regarding screening clients for IPV and choosing referral method. One participant described the importance of clearly explaining various referral methods to clients: “So…people come and choose the method they like…when you explain things clearly it is like the index client is free to choose a method which the contact can use for testing”. Thorough knowledge of available referral methods allows HCWs to actively engage with index clients to discuss strategies to refer contacts in a way that fits their unique confidentiality needs, which was framed as particularly important when IPV is identified as a concern. Multiple participants suggested the use of flipcharts or videos, saying these would save limited HCW time and energy, fill information gaps, and provide clients with a visual aid to supplement the counseling. Others suggested recurring opportunities for training, to continuously “refresh” their ICT knowledge in order to facilitate implementation.

HCW interpersonal skills

In addition, HCWs’ ability to navigate sensitive conversations about HIV was noted as a key facilitator of successful implementation. Interpersonal skills were mentioned as mitigating the role’s day-to-day uncertainty by preparing HCWs to engage with clients, especially newly diagnosed clients: “ I need to counsel them skillfully so that they understand what I mean regardless that they have just tested positive for HIV.”

When discussing strategies to build HCW skills in counseling index clients and tracing contact clients, participants suggested establishing regular opportunities to discuss challenges and share approaches to address these challenges: “ I think that there should be much effort on the [HCWs] doing [ICT]. For example, what do I mean, they should be having a meeting with the facility people to ask what challenges are you facing and how can we end them?”. Another participant further elaborated, saying “We should be able to share experiences with our [colleagues] so that we can all learn from one another. And also, there are other people who are really brilliant at their job. Those people ought to come visit us and see how we are doing. That is very motivating.”

HCW non-stigmatizing attitudes and behaviors

Participants also highlighted the role of empathy and non-judgement in building trust with clients: “ Put yourself in that other person’s shoes. In so doing, the counseling session goes well. Understanding that person, that what is happening to them can also happen to you. ”. Participants viewed trust-building as critical to facilitating client comfort and openness: “if they trust you enough, they will give you the right information.” Further, participants associated HCW assurance of confidentiality with promoting trust and greater information sharing: “ Also assuring them on the issue of confidentiality because confidentiality is a paramount. If there will not be confidentiality then the clients will not reveal.”

HCW sense of purpose

Lastly, several participants reported that a sense of purpose and desire to help people motivated them to overcome the challenges of delivering assisted ICT. One participant said, “ Some of these jobs are a ministry. Counseling is not easy. You just need to tell yourself that you are there to help that person. ” Many seemed to take comfort in the knowledge that their labors, however taxing, would ultimately allow people to know their status, take control of their health, and prevent the spread of HIV. Participants framed the sense of fulfillment from successful ICT implementation as a mitigating factor amidst challenges: “ If [the contact client] has accepted it then I feel that mostly I have achieved the aim of being in the health field…that is why it is appealing to me ”.

Participants described a variety of barriers to assisted ICT implementation, including sensitivities around discussing ICT with clients, privacy concerns, limited time for ICT amid high workloads, poor quality contact information, and logistical obstacles to tracing. These barriers manifested across each step of the process of counseling index clients and tracing contacts. However, participants also identified HCW characteristics and process improvements that can mitigate these barriers.

Further, participants’ descriptions of the assisted ICT process revealed the intimately interconnected nature of factors that influence feasibility of assisted ICT. Sensitivities around HIV, privacy limitations, time constraints, and HCW characteristics all contribute to the extent to which counseling index clients elicits adequate information to facilitate contact tracing. Information quality has implications for HCW capacity, as inadequate information can lead to wasted resources, including HCW time and energy, on contact tracing. The opportunity cost of wasted efforts, which increases as the distance from which the contact client lives from the clinic increases, depletes HCW morale. The resulting acceleration of burnout, which is already fueled by busy workloads and the inherent uncertainty of day-to-day ICT work, further impairs HCW capacity to effectively engage in quality counseling that elicits adequate information from index clients. This interconnectedness suggests that efforts to mitigate barriers at any step of the assisted ICT process may have the potential to ripple across the whole process.

Participants’ descriptions of client confidentiality and privacy concerns, as well as fear of consequences of disclosure, align with previous studies that emphasize stigma as a key barrier to assisted ICT [ 15 , 18 , 19 , 20 , 30 , 31 ] and the overall HIV testing and treatment cascade [ 41 ]. Our findings suggest that anticipated stigma, or the fear of discrimination upon disclosure [ 42 ], drives several key barriers to feasibility of assisted ICT implementation. Previous studies also highlight the key role of HCWs in mitigating barriers related to anticipated stigma; noting the key role of HCW ICT knowledge, interpersonal skills, and non-stigmatizing attitudes/behaviors in securing informed consent from clients for ICT, tailoring the referral strategy to minimize risk to client confidentiality and safety, building trust and rapport with the client, and eliciting accurate contact information from index clients to facilitate contact tracing [ 18 , 19 , 20 , 30 ].

Our findings also reflect previous evidence of logistical challenges related to limited time, space, and resources that can present barriers to feasibility for HCWs [ 18 , 19 , 20 , 30 , 31 ]. Participants in the current study described these logistical challenges as perpetuating HCW burnout, making it harder for them to engage in effective counseling. Cumulative evidence of barriers across different settings (further validated by this study) suggests that assisted ICT implementation may pose greater burden on HCWs than previously thought [ 7 ]. However, our findings also suggest that strategic investment in targeted implementation strategies has the potential to help overcome these feasibility barriers.

In our own work, these findings affirmed the rationale for and informed the development of the blended learning implementation package tested in our trial [ 40 , 43 ]. Findings indicated the need for evidence-based training and support to promote HCW capacity to foster facilitating characteristics. Participants discussed the value of "refresher" opportunities in building knowledge, as well as the value of learning from other’s experiences. The blended learning implementation package balances both needs by providing time for HCWs to master ICT knowledge and skills with a combination of asynchronous, digitally delivered content (which allows for continuous review as a "refresher") and in-person sessions (which allow for sharing, practicing, and feedback). Our findings also highlight the value of flexible referral methods that align with the client’s needs, so our training content includes a detailed description of each referral method process. Further, our training content emphasizes client-centered, non-judgmental counseling as our findings add to cumulative evidence of stigma as a key barrier to assisted ICT implementation [ 41 ].

In addition, participants frequently mentioned informal workarounds currently in use to mitigate barriers or offered up ideas for potential solutions to try. Our blended learning implementation package streamlines these problem-solving processes by offering monthly continuous quality improvement sessions at each facility in our enhanced arm. These sessions allow for structured time to discuss identified barriers, share ideas to mitigate barriers, and develop solutions for sustained process improvement tailored to their specific setting. Initial focus areas for continuous quality improvement discussions include use of space, staffing, allocation of airtime and vehicles, and documentation, which were identified as barriers to feasibility in the current study.

Our study provides a uniquely in-depth examination of HCWs’ experiences implementing assisted ICT, exploring how barriers can manifest and interact with each other at each step of the process to hinder successful implementation. Further, our study has a highly actionable focus on informing development of implementation strategies to support HCWs implementing assisted ICT. Our study also has limitations. Firstly, while our sole focus on HCWs allowed for deeper exploration of assisted ICT from the perspective of those actually implementing it on the ground, this meant that our analysis did not include perspectives of index or contact clients. In addition, we did not conduct sub-group analyses as interpretation of results would be limited by our small sample size.

Assisted ICT has been widely recognized as an evidence-based intervention with high promise to increase PLHIV status awareness [ 5 , 6 , 7 , 10 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 23 , 24 , 26 , 27 , 28 , 29 ], which is important as countries in eastern and southern Africa strive to reach global UNAIDS targets. Study findings support cumulative evidence that HCWs face a variety of feasibility barriers to assisted ICT implementation in the region; further, the study’s uniquely in-depth focus on the experiences of those doing the “assisting” enhances understanding of how these barriers manifest and informs the development of implementation strategies to mitigate these barriers. Maximizing assisted ICT’s potential to increase HIV testing requires equipping HCWs with effective training and support to address and overcome the many feasibility barriers they face in implementation. Findings demonstrate the need for, as well as inform the development of, implementation strategies to mitigate barriers and promote facilitators to feasibility of assisted ICT.

Availability of data and materials

Qualitative data on which this analysis is based, as well as data collection materials and codebooks, are available from the last author upon reasonable request. The interview guide is included as an additional file.

Abbreviations

Acquired Immunodeficiency Syndrome

Antiretroviral Therapy

Health Care Worker

Human Immunodeficiency Virus

HIV Testing Services

Index Case Testing

In-Depth Interview

Intimate Partner Violence

Institutional Review Board

President’s Emergency Plan for HIV/AIDS Relief

People Living With HIV

Joint United Nations Programme on HIV/AIDS

World Health Organization

UNAIDS. Prevailing against pandemics by putting people at the centre. Geneva: UNAIDS; 2020.

Google Scholar  

Frescura L, Godfrey-Faussett P, Feizzadeh AA, El-Sadr W, Syarif O, Ghys PD, et al. Achieving the 95 95 95 targets for all: A pathway to ending AIDS. PLoS One. 2022;17(8):e0272405.

Article   CAS   PubMed   PubMed Central   Google Scholar  

UNAIDS. UNAIDS global AIDS update 2023: The path that ends AIDS. New York: United Nations; 2023.

Book   Google Scholar  

UNAIDS. UNAIDS data 2023. Geneva: Joint United Nations Programme on HIV/AIDS; 2023.

Kahabuka C, Plotkin M, Christensen A, Brown C, Njozi M, Kisendi R, et al. Addressing the first 90: A highly effective partner notification approach reaches previously undiagnosed sexual partners in Tanzania. AIDS Behav. 2017;21(8):2551–60.

Article   PubMed   PubMed Central   Google Scholar  

Lasry A, Medley A, Behel S, Mujawar MI, Cain M, Diekman ST, et al. Scaling up testing for human immunodeficiency virus infection among contacts of index patients - 20 countries, 2016–2018. MMWR Morb Mortal Wkly Rep. 2019;68(21):474–7.

Onovo A, Kalaiwo A, Agweye A, Emmanuel G, Keiser O. Diagnosis and case finding according to key partner risk populations of people living with HIV in Nigeria: A retrospective analysis of community-led index partner testing services. EClinicalMedicine. 2022;43:101265.

World Health Organization (WHO). Guidelines on HIV self-testing and partner notification : supplement to Consolidated guidelines on HIV testing services. 2016. https://apps.who.int/iris/bitstream/handle/10665/251655/9789241549868-eng.pdf?sequence=1 . Accessed 19 Apr 2024.

Watts H. Why PEPFAR is going all in on partner notification services. 2019. https://programme.ias2019.org/PAGMaterial/PPT/1934_117/Why%20PEPFAR%20is%20all%20in%20for%20PNS%2007192019%20rev.pptx . Accessed 19 Apr 2024.

Dalal S, Johnson C, Fonner V, Kennedy CE, Siegfried N, Figueroa C, et al. Improving HIV test uptake and case finding with assisted partner notification services. AIDS. 2017;31(13):1867–76.

Article   PubMed   Google Scholar  

Mathews C, Coetzee N, Zwarenstein M, Lombard C, Guttmacher S, Oxman A, et al. A systematic review of strategies for partner notification for sexually transmitted diseases, including HIV/AIDS. Int J STD AIDS. 2002;13(5):285–300.

Hogben M, McNally T, McPheeters M, Hutchinson AB. The effectiveness of HIV partner counseling and referral services in increasing identification of HIV-positive individuals a systematic review. Am J Prev Med. 2007;33(2 Suppl):S89-100.

Brown LB, Miller WC, Kamanga G, Nyirenda N, Mmodzi P, Pettifor A, et al. HIV partner notification is effective and feasible in sub-Saharan Africa: opportunities for HIV treatment and prevention. J Acquir Immune Defic Syndr. 2011;56(5):437–42.

Sharma M, Ying R, Tarr G, Barnabas R. Systematic review and meta-analysis of community and facility-based HIV testing to address linkage to care gaps in sub-Saharan Africa. Nature. 2015;528(7580):S77-85.

Edosa M, Merdassa E, Turi E. Acceptance of index case HIV testing and its associated factors among HIV/AIDS Clients on ART follow-up in West Ethiopia: A multi-centered facility-based cross-sectional study. HIV AIDS (Auckl). 2022;14:451–60.

PubMed   Google Scholar  

Williams D, MacKellar D, Dlamini M, Byrd J, Dube L, Mndzebele P, et al. HIV testing and ART initiation among partners, family members, and high-risk associates of index clients participating in the CommLink linkage case management program, Eswatini, 2016–2018. PLoS ONE. 2021;16(12):e0261605.

Remera E, Nsanzimana S, Chammartin F, Semakula M, Rwibasira GN, Malamba SS, et al. Brief report: Active HIV case finding in the city of Kigali, Rwanda: Assessment of voluntary assisted partner notification modalities to detect undiagnosed HIV infections. J Acquir Immune Defic Syndr. 2022;89(4):423–7.

Article   CAS   PubMed   Google Scholar  

Quinn C, Nakyanjo N, Ddaaki W, Burke VM, Hutchinson N, Kagaayi J, et al. HIV partner notification values and preferences among sex workers, fishermen, and mainland community members in Rakai, Uganda: A qualitative study. AIDS Behav. 2018;22(10):3407–16.

Monroe-Wise A, Maingi Mutiti P, Kimani H, Moraa H, Bukusi DE, Farquhar C. Assisted partner notification services for patients receiving HIV care and treatment in an HIV clinic in Nairobi, Kenya: a qualitative assessment of barriers and opportunities for scale-up. J Int AIDS Soc. 2019;22 Suppl 3(Suppl Suppl 3):e25315.

Liu W, Wamuti BM, Owuor M, Lagat H, Kariithi E, Obong’o C, et al. “It is a process” - a qualitative evaluation of provider acceptability of HIV assisted partner services in western Kenya: experiences, challenges, and facilitators. BMC Health Serv Res. 2022;22(1):616.

Myers RS, Feldacker C, Cesar F, Paredes Z, Augusto G, Muluana C, et al. Acceptability and effectiveness of assisted human immunodeficiency virus partner services in Mozambique: Results from a pilot program in a public. Urban Clinic Sex Transm Dis. 2016;43(11):690–5.

Rosenberg NE, Mtande TK, Saidi F, Stanley C, Jere E, Paile L, et al. Recruiting male partners for couple HIV testing and counselling in Malawi’s option B+ programme: an unblinded randomised controlled trial. Lancet HIV. 2015;2(11):e483–91.

Mahachi N, Muchedzi A, Tafuma TA, Mawora P, Kariuki L, Semo BW, et al. Sustained high HIV case-finding through index testing and partner notification services: experiences from three provinces in Zimbabwe. J Int AIDS Soc. 2019;22 Suppl 3(Suppl Suppl 3):e25321.

Cherutich P, Golden MR, Wamuti B, Richardson BA, Asbjornsdottir KH, Otieno FA, et al. Assisted partner services for HIV in Kenya: a cluster randomised controlled trial. Lancet HIV. 2017;4(2):e74–82.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76.

Kamanga G, Brown L, Jawati P, Chiwanda D, Nyirenda N. Maximizing HIV partner notification opportunities for index patients and their sexual partners in Malawi. Malawi Med J. 2015;27(4):140–4.

Rutstein SE, Brown LB, Biddle AK, Wheeler SB, Kamanga G, Mmodzi P, et al. Cost-effectiveness of provider-based HIV partner notification in urban Malawi. Health Policy Plan. 2014;29(1):115–26.

Wamuti BM, Welty T, Nambu W, Chimoun FT, Shields R, Golden MR, et al. Low risk of social harms in an HIV assisted partner services programme in Cameroon. J Int AIDS Soc. 2019;22 Suppl 3(Suppl Suppl 3):e25308.

Henley C, Forgwei G, Welty T, Golden M, Adimora A, Shields R, et al. Scale-up and case-finding effectiveness of an HIV partner services program in Cameroon: an innovative HIV prevention intervention for developing countries. Sex Transm Dis. 2013;40(12):909–14.

Klabbers RE, Muwonge TR, Ayikobua E, Izizinga D, Bassett IV, Kambugu A, et al. Health worker perspectives on barriers and facilitators of assisted partner notification for HIV for refugees and Ugandan nationals: A mixed methods study in West Nile Uganda. AIDS Behav. 2021;25(10):3206–22.

Mugisha N, Tirera F, Coulibaly-Kouyate N, Aguie W, He Y, Kemper K, et al. Implementation process and challenges of index testing in Cote d’Ivoire from healthcare workers’ perspectives. PLoS One. 2023;18(2):e0280623.

Rosenberg NE, Tembo TA, Simon KR, Mollan K, Rutstein SE, Mwapasa V, et al. Development of a Blended Learning Approach to Delivering HIV-Assisted Contact Tracing in Malawi: Applied Theory and Formative Research. JMIR Form Res. 2022;6(4):e32899.

Government of Malawi National Statistical Office. 2018 Malawi population and housing census : main report. 2019.  https://malawi.unfpa.org/sites/default/files/resource-pdf/2018%20Malawi%20Population%20and%20Housing%20Census%20Main%20Report%20%281%29.pdf . Accessed 19 April 2024. 

Wolock TM, Flaxman S, Chimpandule T, Mbiriyawanda S, Jahn A, Nyirenda R, et al. Subnational HIV incidence trends in Malawi: large, heterogeneous declines across space. medRxiv (PREPRINT). 2023. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9915821/ . Accessed 19 Apr 2024.

World Health Organization (WHO). Medical doctors (per 10,000). 2020. https://www.who.int/data/gho/data/indicators/indicator-details/GHO/medical-doctors-(per-10-000-population . Accessed 19 Apr 2024.

Flick RJ, Simon KR, Nyirenda R, Namachapa K, Hosseinipour MC, Schooley A, et al. The HIV diagnostic assistant: early findings from a novel HIV testing cadre in Malawi. AIDS. 2019;33(7):1215–24.

Kim MH, Ahmed S, Buck WC, Preidis GA, Hosseinipour MC, Bhalakia A, et al. The Tingathe programme: a pilot intervention using community health workers to create a continuum of care in the prevention of mother to child transmission of HIV (PMTCT) cascade of services in Malawi. J Int AIDS Soc. 2012;15(Suppl 2):17389.

Simon KR, Hartig M, Abrams EJ, Wetzel E, Ahmed S, Chester E, et al. The Tingathe Surge: a multi-strategy approach to accelerate HIV case finding in Malawi. Public Health Action. 2019;9(3):128–34.

Ahmed S, Kim MH, Dave AC, Sabelli R, Kanjelo K, Preidis GA, et al. Improved identification and enrolment into care of HIV-exposed and -infected infants and children following a community health worker intervention in Lilongwe, Malawi. J Int AIDS Soc. 2015;18(1):19305.

Tembo TA, Mollan K, Simon K, Rutstein S, Chitani MJ, Saha PT, et al. Does a blended learning implementation package enhance HIV index case testing in Malawi? A protocol for a cluster randomised controlled trial. BMJ Open. 2024;14(1):e077706.

Nyblade L, Mingkwan P, Stockton MA. Stigma reduction: an essential ingredient to ending AIDS by 2030. Lancet HIV. 2021;8(2):e106–13.

Nyblade L, Stockton M, Nyato D, Wamoyi J. Perceived, anticipated and experienced stigma: exploring manifestations and implications for young people’s sexual and reproductive health and access to care in North-Western Tanzania. Cult Health Sex. 2017;19(10):1092–107.

Tembo TA, Simon KR, Kim MH, Chikoti C, Huffstetler HE, Ahmed S, et al. Pilot-Testing a Blended Learning Package for Health Care Workers to Improve Index Testing Services in Southern Malawi: An Implementation Science Study. J Acquir Immune Defic Syndr. 2021;88(5):470–6.

Download references

Acknowledgements

We are grateful to the Malawian health care workers who shared their experiences through in-depth interviews, as well as to the study team members in Malawi and the United States for their contributions.

Research reported in this publication was funded by the National Institutes of Health (R01 MH124526) with support from the University of North Carolina at Chapel Hill Center for AIDS Research (P30 AI50410) and the Fogarty International Center of the National Institutes of Health (D43 TW010060 and R01 MH115793-04). The funders had no role in trial design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and affiliations.

RTI International, Research Triangle Park, NC, USA

Caroline J. Meek

Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

Caroline J. Meek, Milenka Jean-Baptiste, Jiayu Wang, Clare Barrington, Vivian F. Go & Nora E. Rosenberg

Kamuzu University of Health Sciences, Blantyre, Malawi

Tiwonge E. Mbeya Munkhondya

Baylor College of Medicine Children’s Foundation, Lilongwe, Malawi

Mtisunge Mphande, Tapiwa A. Tembo, Mike Chitani, Dhrutika Vansia, Caroline Kumbuyo, Katherine R. Simon & Maria H. Kim

Department of Medicine, Division of Infectious Diseases, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

Sarah E. Rutstein

You can also search for this author in PubMed   Google Scholar

Contributions

TAT, KRS, SER, MHK, VFG, and NER contributed to overall study conceptualization, with CJM, CB, and NER leading conceptualization of the analysis presented in this study. Material preparation and data collection were performed by TEMM, MM, TAT, MC, and CK. Analysis was led by CJM with support from MJB and DV. The first draft of the manuscript was written by CJM with consultation from NER, TEMM, MM, TAT, MJB, and DV. JW provided quantitative analysis support for participant characteristics. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Caroline J. Meek .

Ethics declarations

Ethics approval and consent to participate.

Ethical clearance was provided by the Malawi National Health Science Research Committee (NHSRC; #20/06/2566), University of North Carolina Institution Review Board (UNC IRB; #20–1810) and the Baylor College of Medicine institutional review board (Baylor IRB; H-48800). The procedures used in this study adhere to the tenets of the Declaration of Helsinki. Written informed consent for participation was obtained from all study participants prior to enrollment in the parent study. Interviewers also engaged in informal verbal discussion of consent immediately ahead of in-depth interviews.

Consent for publication

Not applicable. No identifying information is included in the manuscript.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Meek, C.J., Munkhondya, T.E.M., Mphande, M. et al. Examining the feasibility of assisted index case testing for HIV case-finding: a qualitative analysis of barriers and facilitators to implementation in Malawi. BMC Health Serv Res 24 , 606 (2024). https://doi.org/10.1186/s12913-024-10988-z

Download citation

Received : 31 August 2023

Accepted : 12 April 2024

Published : 09 May 2024

DOI : https://doi.org/10.1186/s12913-024-10988-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• HIV testing and counseling
• Index case testing
• Assisted partner notification services
• Implementation science
• Health care workers

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Experience new growth possibilities with Microsoft Advertising today >

Japan delivers high-value audience reach for trivago

Trivago logo

Expanding to new horizons

When Microsoft Advertising launched in Japan in May 2022, trivago was keen to be one of the first advertisers to run search and native campaigns targeting Microsoft Advertising’s high-quality audience.

In Japan, Microsoft has strong rates of Edge browser usage, Microsoft Bing, and high Windows PC adoption. By working directly with Japanese agencies and digital marketers, Microsoft Advertising can empower more marketers to reach over 1 billion audiences via search and native solutions targeting high-quality audiences across Microsoft properties, globally.

For trivago , having the opportunity to reach new audiences around the world is key. The metasearch allows travelers to make informed decisions by personalizing their search for accommodations and providing them with access to more than 5 million hotels and other types of accommodation in over 190 countries.

When trivago’s Head of Performance Marketing, Bruno Frangen, heard Microsoft Advertising was expanding into Japan, he didn't want to miss the opportunity to reach new audiences. "Considering Microsoft Advertising’s rich data and knowing that the quality of trivago users in Japan is very high, we thought it was a great opportunity," explains Bruno.

"There's a lot of value in the market. We were eager to see what we could get out of Japan".

“We can’t ignore the potential of Microsoft Advertising. It makes a difference and it makes sense for us to be here [in Japan]”

— Bruno Frangen, Head of Performance Marketing, trivago

How trivago maximized audiences in Japan

Getting started was easy for trivago. Thanks to its previous experience in the market, the company already knew what users were searching for in Japan and which destinations were the most popular.

"The whole process was very smooth," says Bruno. "We used existing campaign content and our knowledge of operating in Japan with other search engines to quickly create new campaigns in the Microsoft Advertising platform."

They started out with the basics: Text ads. And to avoid having to manually monitor performance, trivago decided to try the Target cost per acquisition CPA automated bidding strategy to save time and improve ad efficiency. Once everything was set up, the volume started to come in.

Higher conversion at a cheaper cost

The ratio of clicks that converted to actual bookings was 20% higher on Microsoft Advertising, while it was 25% cheaper in cost per click (CPC) from June through July 2022, despite lower volume compared to competitors.

“Even if the volume is smaller than Google or Yahoo, with this amount of traffic, we just can’t ignore the potential of Microsoft Advertising,” said Bruno. “It makes a difference, and it makes sense for us to be here”.

Now, trivago achieves more booking value with the same investment than other search platforms.

“At the moment, we’re having around 50% less price of buying the booking value compared to others,” admits Bruno.

The high quality of the Microsoft Advertising audience has been a differential factor for trivago.

“The type of traffic we’re getting from Microsoft is usually desktop heavy, and they come with higher likelihood of making the purchase compared to the mobile users”.

"Considering Microsoft Advertising’s rich data and knowing that the quality of trivago users in Japan is very high, we thought it was a great opportunity.”

See a summary

Download the infographic to take the trivago story with you.

Stay informed

Sign up for the Microsoft Advertising Insider newsletter to keep up with the latest insights, product news, tips and tricks, thought leadership, customer case studies, and resources.

Recommended for you

New bid strategies for audience ads and other product updates.

Learn about our new bid strategies for Audience ads: Maximize Conversions and Target CPA.

November 07, 2023

The impact of highly visual ads done right

Through highly visual Multimedia Ads, Reservations.com was able to scale their reach to more travelers with a strict goal for their ROAS.

January 11, 2023

HDFC Life runs an awareness drive during the India Premier League

Learn how HDFC Life ran a branding campaign with Microsoft Advertising to attract Indian Premier League viewers and drive brand awareness.

November 14, 2022