research paper software testing

software testing Recently Published Documents

Total documents.

• Latest Documents
• Most Cited Documents
• Contributed Authors
• Related Sources
• Related Keywords

Combining Learning and Engagement Strategies in a Software Testing Learning Environment

There continues to be an increase in enrollments in various computing programs at academic institutions due to many job opportunities available in the information, communication, and technology sectors. This enrollment surge has presented several challenges in many Computer Science (CS), Information Technology (IT), and Software Engineering (SE) programs at universities and colleges. One such challenge is that many instructors in CS/IT/SE programs continue to use learning approaches that are not learner centered and therefore are not adequately preparing students to be proficient in the ever-changing computing industry. To mitigate this challenge, instructors need to use evidence-based pedagogical approaches, e.g., active learning, to improve student learning and engagement in the classroom and equip students with the skills necessary to be lifelong learners. This article presents an approach that combines learning and engagement strategies (LESs) in learning environments using different teaching modalities to improve student learning and engagement. We describe how LESs are integrated into face-to-face (F2F) and online class activities. The LESs currently used are collaborative learning , gamification , problem-based learning , and social interaction . We describe an approach used to quantify each LES used during class activities based on a set of characteristics for LESs and the traditional lecture-style pedagogical approaches. To demonstrate the impact of using LESs in F2F class activities, we report on a study conducted over seven semesters in a software testing class at a large urban minority serving institution. The study uses a posttest-only study design, the scores of two midterm exams, and approximate class times dedicated to each LES and traditional lecture style to quantify their usage in a face-to-face software testing class. The study results showed that increasing the time dedicated to collaborative learning, gamification, and social interaction and decreasing the traditional lecture-style approach resulted in a statistically significant improvement in student learning, as reflected in the exam scores.

Enhancing Search-based Testing with Testability Transformations for Existing APIs

Search-based software testing (SBST) has been shown to be an effective technique to generate test cases automatically. Its effectiveness strongly depends on the guidance of the fitness function. Unfortunately, a common issue in SBST is the so-called flag problem , where the fitness landscape presents a plateau that provides no guidance to the search. In this article, we provide a series of novel testability transformations aimed at providing guidance in the context of commonly used API calls (e.g., strings that need to be converted into valid date/time objects). We also provide specific transformations aimed at helping the testing of REST Web Services. We implemented our novel techniques as an extension to EvoMaster , an SBST tool that generates system-level test cases. Experiments on nine open-source REST web services, as well as an industrial web service, show that our novel techniques improve performance significantly.

A Survey of Flaky Tests

Tests that fail inconsistently, without changes to the code under test, are described as flaky . Flaky tests do not give a clear indication of the presence of software bugs and thus limit the reliability of the test suites that contain them. A recent survey of software developers found that 59% claimed to deal with flaky tests on a monthly, weekly, or daily basis. As well as being detrimental to developers, flaky tests have also been shown to limit the applicability of useful techniques in software testing research. In general, one can think of flaky tests as being a threat to the validity of any methodology that assumes the outcome of a test only depends on the source code it covers. In this article, we systematically survey the body of literature relevant to flaky test research, amounting to 76 papers. We split our analysis into four parts: addressing the causes of flaky tests, their costs and consequences, detection strategies, and approaches for their mitigation and repair. Our findings and their implications have consequences for how the software-testing community deals with test flakiness, pertinent to practitioners and of interest to those wanting to familiarize themselves with the research area.

Test Suite Optimization Using Firefly and Genetic Algorithm

Software testing is essential for providing error-free software. It is a well-known fact that software testing is responsible for at least 50% of the total development cost. Therefore, it is necessary to automate and optimize the testing processes. Search-based software engineering is a discipline mainly focussed on automation and optimization of various software engineering processes including software testing. In this article, a novel approach of hybrid firefly and a genetic algorithm is applied for test data generation and selection in regression testing environment. A case study is used along with an empirical evaluation for the proposed approach. Results show that the hybrid approach performs well on various parameters that have been selected in the experiments.

Machine Learning Model to Predict Automated Testing Adoption

Software testing is an activity conducted to test the software under test. It has two approaches: manual testing and automation testing. Automation testing is an approach of software testing in which programming scripts are written to automate the process of testing. There are some software development projects under development phase for which automated testing is suitable to use and other requires manual testing. It depends on factors like project requirements nature, team which is working on the project, technology on which software is developing and intended audience that may influence the suitability of automated testing for certain software development project. In this paper we have developed machine learning model for prediction of automated testing adoption. We have used chi-square test for finding factors’ correlation and PART classifier for model development. Accuracy of our proposed model is 93.1624%.

Metaheuristic Techniques for Test Case Generation

The primary objective of software testing is to locate bugs as many as possible in software by using an optimum set of test cases. Optimum set of test cases are obtained by selection procedure which can be viewed as an optimization problem. So metaheuristic optimizing (searching) techniques have been immensely used to automate software testing task. The application of metaheuristic searching techniques in software testing is termed as Search Based Testing. Non-redundant, reliable and optimized test cases can be generated by the search based testing with less effort and time. This article presents a systematic review on several meta heuristic techniques like Genetic Algorithms, Particle Swarm optimization, Ant Colony Optimization, Bee Colony optimization, Cuckoo Searches, Tabu Searches and some modified version of these algorithms used for test case generation. The authors also provide one framework, showing the advantages, limitations and future scope or gap of these research works which will help in further research on these works.

Software Testing Under Agile, Scrum, and DevOps

The adoption of agility at a large scale often requires the integration of agile and non-agile development practices into hybrid software development and delivery environment. This chapter addresses software testing related issues for Agile software application development. Currently, the umbrella of Agile methodologies (e.g. Scrum, Extreme Programming, Development and Operations – i.e., DevOps) have become the preferred tools for modern software development. These methodologies emphasize iterative and incremental development, where both the requirements and solutions evolve through the collaboration between cross-functional teams. The success of such practices relies on the quality result of each stage of development, obtained through rigorous testing. This chapter introduces the principles of software testing within the context of Scrum/DevOps based software development lifecycle.

Quality Assurance Issues for Big Data Applications in Supply Chain Management

Heterogeneous data types, widely distributed data sources, huge data volumes, and large-scale business-alliance partners describe typical global supply chain operational environments. Mobile and wireless technologies are putting an extra layer of data source in this technology-enriched supply chain operation. This environment also needs to provide access to data anywhere, anytime to its end-users. This new type of data set originating from the global retail supply chain is commonly known as big data because of its huge volume, resulting from the velocity with which it arrives in the global retail business environment. Such environments empower and necessitate decision makers to act or react quicker to all decision tasks. Academics and practitioners are researching and building the next generation of big-data-based application software systems. This new generation of software applications is based on complex data analysis algorithms (i.e., on data that does not adhere to standard relational data models). The traditional software testing methods are insufficient for big-data-based applications. Testing big-data-based applications is one of the biggest challenges faced by modern software design and development communities because of lack of knowledge on what to test and how much data to test. Big-data-based applications developers have been facing a daunting task in defining the best strategies for structured and unstructured data validation, setting up an optimal test environment, and working with non-relational databases testing approaches. This chapter focuses on big-data-based software testing and quality-assurance-related issues in the context of Hadoop, an open source framework. It includes discussion about several challenges with respect to massively parallel data generation from multiple sources, testing methods for validation of pre-Hadoop processing, software application quality factors, and some of the software testing mechanisms for this new breed of applications

Use of Qualitative Research to Generate a Function for Finding the Unit Cost of Software Test Cases

In this article, we demonstrate a novel use of case research to generate an empirical function through qualitative generalization. This innovative technique applies interpretive case analysis to the problem of defining and generalizing an empirical cost function for test cases through qualitative interaction with an industry cohort of subject matter experts involved in software testing at leading technology companies. While the technique is fully generalizable, this article demonstrates this technique with an example taken from the important field of software testing. The huge amount of software development conducted in today's world makes taking its cost into account imperative. While software testing is a critical aspect of the software development process, little attention has been paid to the cost of testing code, and specifically to the cost of test cases, in comparison to the cost of developing code. Our research fills the gap by providing a function for estimating the cost of test cases.

Framework for Reusable Test Case Generation in Software Systems Testing

Agile methodologies have become the preferred choice for modern software development. These methods focus on iterative and incremental development, where both requirements and solutions develop through collaboration among cross-functional software development teams. The success of a software system is based on the quality result of each stage of development with proper test practice. A software test ontology should represent the required software test knowledge in the context of the software tester. Reusing test cases is an effective way to improve the testing of software. The workload of a software tester for test-case generation can be improved, previous software testing experience can be shared, and test efficiency can be increased by automating software testing. In this chapter, the authors introduce a software testing framework (STF) that uses rule-based reasoning (RBR), case-based reasoning (CBR), and ontology-based semantic similarity assessment to retrieve the test cases from the case library. Finally, experimental results are used to illustrate some of the features of the framework.

Export Citation Format

Share document.

Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.

• Qlik Community
• Member Articles
• Comparing Qlik Analytics and R Software for Accura...
• Edit Document
• Move Document
• Delete Document
• Subscribe to RSS Feed
• Mark as New
• Mark as Read
• Printer Friendly Page
• Report Inappropriate Content

Comparing Qlik Analytics and R Software for Accurate Two-Sample T-Tests Research Paper

May 24, 2024 2:46:29 PM

igoralcantara

Client Managed

Software Testing Techniques: A Literature Review

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

A systematic literature review on software security testing using metaheuristics

• Published: 23 May 2024
• Volume 31 , article number  44 , ( 2024 )

Cite this article

• Fatma Ahsan 1 &
• Faisal Anwer 1  

23 Accesses

Explore all metrics

The security of an application is critical for its success, as breaches cause loss for organizations and individuals. Search-based software security testing (SBSST) is the field that utilizes metaheuristics to generate test cases for the software testing for some pre-specified security test adequacy criteria This paper conducts a systematic literature review to compare metaheuristics and fitness functions used in software security testing, exploring their distinctive capabilities and impact on vulnerability detection and code coverage. The aim is to provide insights for fortifying software systems against emerging threats in the rapidly evolving technological landscape. This paper examines how search-based algorithms have been explored in the context of code coverage and software security testing. Moreover, the study highlights different metaheuristics and fitness functions for security testing and code coverage. This paper follows the standard guidelines from Kitchenham to conduct SLR and obtained 122 primary studies related to SBSST after a multi-stage selection process. The papers were from different sources journals, conference proceedings, workshops, summits, and researchers’ webpages published between 2001 and 2022. The outcomes demonstrate that the main tackled vulnerabilities using metaheuristics are XSS, SQLI, program crash, and XMLI. The findings have suggested several areas for future research directions, including detecting server-side request forgery and security testing of third-party components. Moreover, new metaheuristics must also need to be explored to detect security vulnerabilities that are still unexplored or explored significantly less. Furthermore, metaheuristics can be combined with machine learning and reinforcement learning techniques for better results. Some metaheuristics can be designed by looking at the complexity of security testing and exploiting more fitness functions related to detecting different vulnerabilities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Systematic literature review on software quality for AI-based software

Metaheuristics: a comprehensive overview and classification along with bibliometric analysis

Testing machine learning based systems: a systematic mapping

Abbreviations.

Firefly algorithm

Cuckoo search

Genetic algorithm

Simulated annealing

Grammatical evolution

Genetic programming

Test object

Hill climbing

Memetic algorithm

Harmony search

Evolutionary programming

• Evolutionary algorithm

Bat algorithm

Randomized algorithm

Evolutionary strategies

Differential evolution

Greedy search

Local Search

Null pointer exception

Cross site scripting

Standard genetic algorithm

Co-evolutionary algorithm

Hybrid genetic algorithm

Particle swarm optimization

Artificial bee colony optimization

Many independent objective

Hill climbing algorithm

Denial of service

Domain object model

Ant colony optimization

Improved genetic algorithm

Hill climbing using Korel’s AVM

K medoids algorithm

Hybrid evolutionary algorithm

Real-coded genetic algorithm

Whole test suite

Gene expression programming

Weighted genetic algorithm

Artificial bee colony algorithm

Memetic genetic algorithm

Structured query language injection

Extensible markup language injection

Multi-objective genetic algorithm

Dynamic principal component analysis

Multi-objective simulated annealing

Search-based software testing

Search-based software engineering

Common vulnerability scoring system

Co-operative co-evolutionary algorithm

Search-based software security testing

Multi-objective evolutionary search adaptive random testing

Fixed-sized candidate-set adaptive random testing

Collaborative co-evolutionary contract-driven algorithm

Multi-objective evolutionary algorithm based on decomposition

Multi-objective co-operative co-evolutionary algorithm

Evolutionary adaptive random testing algorithm

Dynamic multi-objective sorting algorithm

Non-dominated sorting genetic algorithm

Vector evaluated genetic algorithm

Niched pareto genetic algorithm

Afshan, S., McMinn, P., Stevenson, M.: Evolving readable string test inputs using a natural language model to reduce human oracle cost. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, pp. 352–361. IEEE (2013)

Afzal, W., Torkar, R., Feldt, R.: A systematic review of search-based testing for non-functional system properties. Inf. Softw. Technol. 51 (6), 957–976 (2009)

Article   Google Scholar  

Ahmed, M.A., Ali, F.: Multiple-path testing for cross site scripting using genetic algorithms. J. Syst. Architect. 64 , 50–62 (2016)

Ahsan, F., Anwer, F.: A critical review on search-based security testing of programs. Comput. Intell. Select Proc. InCITe 2022 , 207–225 (2023)

Almulla, H., Gay, G.: Learning how to search: generating effective test cases through adaptive fitness function selection. Empir. Softw. Eng. 27 (2), 1–62 (2022)

Alshahwan, N., Harman, M.: Automated web application testing using search based software engineering. In: 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011), pp. 3–12. IEEE (2011)

Alyasiri, H.: Evolving rules for detecting cross-site scripting attacks using genetic programming. In: International Conference on Advances in Cyber Security, pp. 642–656. Springer (2020)

Anand, S., Burke, E.K., Chen, T.Y., Clark, J., Cohen, M.B., Grieskamp, W., Harman, M., Harrold, M.J., McMinn, P., Bertolino, A., et al.: An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86 (8), 1978–2001 (2013)

Anas, M., Imam, R., Anwer, F.: Elliptic curve cryptography in cloud security: a survey. In: 2022 12th International Conference on Cloud Computing, Data Science and Engineering (Confluence), pp. 112–117. IEEE (2022)

Andrews, A., Boukhris, S., Elakeili, S.: Fail-safe testing of web applications. In: 2014 23rd Australian Software Engineering Conference, pp. 200–209. IEEE (2014)

Anjum, M.S., Ryan, C.: Seeding grammars in grammatical evolution to improve search-based software testing. SN Comput. Sci. 2 (4), 1–19 (2021)

Anwer, F., Nazir, M., Mustafa, K.: Testing program for security using symbolic execution and exception injection. Indian J. Sci. Technol. 9 , 19 (2016)

Google Scholar  

Anwer, F., Nazir, M., Mustafa, K.: Safety and security framework for exception handling in concurrent programming. In: 2013 Third International Conference on Advances in Computing and Communications, pp. 308–311. IEEE (2013)

Anwer, F., Nazir, M., Mustafa, K.: Automatic testing of inconsistency caused by improper error handling: a safety and security perspective. In: Proceedings of the 2014 International Conference on Information and Communication Technology for Competitive Strategies, pp. 1–5 (2014)

Anwer, F., Nazir, M., Mustafa, K.: Security testing. Trends in Software Testing, pp. 35–66 (2017)

Anwer, F., Nazir, M., Mustafa, K.: Testing program crash based on search based testing and exception injection. In: International Conference on Security & Privacy, pp. 275–285. Springer (2019)

Arcuri, A.: Test suite generation with the many independent objective (MIO) algorithm. Inf. Softw. Technol. 104 , 195–206 (2018)

Arcuri, A.: Restful API automated test case generation with EvoMaster. ACM Trans. Softw. Eng. Methodol. 28 (1), 1–37 (2019)

Article   MathSciNet   Google Scholar  

Arcuri, A., Galeotti, J.P.: Handling SQL databases in automated system test generation. ACM Trans. Softw. Eng. Methodol. 29 (4), 1–31 (2020)

Arcuri, A., Galeotti, J.P.: Enhancing search-based testing with testability transformations for existing APIS. ACM Trans. Softw. Eng. Methodol. 31 (1), 1–34 (2021)

Arcuri, A.: Restful API automated test case generation. In: 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS), pp. 9–20. IEEE (2017)

Arcuri, A.: Evomaster: Evolutionary multi-context automated system test generation. In: 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), pp. 394–397. IEEE (2018a)

Avancini, A., Ceccato, M.: Comparison and integration of genetic algorithms and dynamic symbolic execution for security testing of cross-site scripting vulnerabilities. Inf. Softw. Technol. 55 (12), 2209–2222 (2013)

Avancini, A.: Security testing of web applications: a research plan. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 1491–1494. IEEE (2012)

Avancini, A. and Ceccato, M.: Towards security testing with taint analysis and genetic algorithms. In:Proceedings of the 2010 ICSE Workshop on Software Engineering for Secure Systems, pp. 65–71 (2010)

Avancini, A., Ceccato, M.: Security testing of web applications: A search-based approach for cross-site scripting vulnerabilities. In: 2011 IEEE 11th International Working Conference on Source Code Analysis and Manipulation, pp. 85–94. IEEE (2011)

Avancini, A., Ceccato, M.: Grammar based oracle for security testing of web applications. In: 2012 7th International Workshop on Automation of Software Test (AST), pp. 15–21. IEEE (2012)

Aziz, B., Bader, M., Hippolyte, C.: Search-based sql injection attacks testing using genetic programming. In: European Conference on Genetic Programming, pp. 183–198. Springer (2016)

Balera, J.M., de Santiago Júnior, V.A.: A systematic mapping addressing hyper-heuristics within search-based software testing. Inf. Softw. Technol. 114 , 176–189 (2019)

Baluda, M.: Evose: evolutionary symbolic execution. In: Proceedings of the 6th International Workshop on Automating Test Case Design, Selection and Evaluation, pp. 16–19 (2015)

Baresel, A., Pohlheim, H., Sadeghipour, S.: Structural and functional sequence test of dynamic and state-based software with evolutionary algorithms. In: Genetic and Evolutionary Computation Conference, pp. 2428–2441. Springer (2003)

Baresel, A., Sthamer, H.: Evolutionary testing of flag conditions. In: Genetic and Evolutionary Computation Conference, pp. 2442–2454. Springer (2003)

Bejo, S. D., Assefa, B. G., Mohapatra, S. K.: Backip: Mutation based test data generation using hybrid approach. In: 2021 International Conference on Information and Communication Technology for Development for Africa (ICT4DA), pp. 178–183. IEEE (2021)

Benito-Parejo, M., Merayo, M. G.: Using genetic algorithms to select test cases for finite state machines with timeouts. In: 2021 IEEE Congress on Evolutionary Computation (CEC), pp. 2403–2410. IEEE (2021)

Bhattacharya, N., Sakti, A., Antoniol, G., Guéhéneuc, Y.-G., Pesant, G.: Divide-by-zero exception raising via branch coverage. In: International Symposium on Search Based Software Engineering, pp. 204–218. Springer (2011)

Boopathi, M., Sujatha, R., Kumar, C.S., Narasimman, S., Rajan, A.: Markov approach for quantifying the software code coverage using genetic algorithm in software testing. Int. J. Bio-Inspired Comput. 14 (1), 27–45 (2019)

Bottaci, L.: Instrumenting programs with flag variables for test data search by genetic algorithm. In: Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation, pp. 1337–1342 (2002)

CWE - Common Weakness Enumeration. https://cwe.mitre.org/

Cao, Y., Hu, C., Li, L.: An approach to generate software test data for a specific path automatically with genetic algorithm. In: 2009 8th International Conference on Reliability, Maintainability and Safety, pp. 888–892. IEEE (2009a)

Cao, Y., Hu, C., Li, L.: Search-based multi-paths test data generation for structure-oriented testing. In: Proceedings of the first ACM/SIGEVO Summit on Genetic and Evolutionary Computation, pp. 25–32 (2009b)

Castelein, J., Aniche, M., Soltani, M., Panichella, A., van Deursen, A.: Search-based test data generation for SQL queries. In: Proceedings of the 40th International Conference on Software Engineering, pp. 1220–1230 (2018)

Ceccato, M., Nguyen, C. D., Appelt, D., Briand, L. C.: Sofia: An automated security oracle for black-box testing of SQL-injection vulnerabilities. In: 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 167–177. IEEE (2016)

Chang, B.-M., Choi, K.: A review on exception analysis. Inf. Softw. Technol. 77 , 1–16 (2016)

Charmchi, M. R. H., Cami, B. R.: Paths-oriented test data generation using genetic algorithm. In: 2021 12th International Conference on Information and Knowledge Technology (IKT), pp. 157–162. IEEE (2021)

Costa, G., Valenza, A.: Why Charles can pen-test: an evolutionary approach to vulnerability testing (2020). arXiv preprint https://arxiv.org/abs/2011.13213

Cui, B., Liang, X., Wang, J.: The study on integer overflow vulnerability detection in binary executables based upon genetic algorithm. In: Foundations of Intelligent Systems, pp. 259–266. Springer (2011)

Dass, S., Namin, A. S.: Evolutionary algorithms for vulnerability coverage. In: 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC), pp. 1795–1801. IEEE (2020a)

Dass, S., Namin, A. S.: Vulnerability coverage as an adequacy testing criterion. arXiv preprint https://arxiv.org/abs/2006.08606 (2020b)

Dass, S., Namin, A. S.: Vulnerability coverage for adequacy security testing. In: Proceedings of the 35th Annual ACM Symposium on Applied Computing, pp. 540–543 (2020c)

Dass, S., Namin, A. S.: Vulnerability coverage for secure configuration (2020d). arXiv preprint https://arxiv.org/abs/2006.08604

de Almeida Biolchini, J.C., Mian, P.G., Natali, A.C.C., Conte, T.U., Travassos, G.H.: Scientific research ontology to support systematic review in software engineering. Adv. Eng. Inform. 21 (2), 133–151 (2007)

Del Grosso, C., Antoniol, G., Di Penta, M.: An evolutionary testing approach to detect buffer overflow. In: Student Paper Proceedings of the International Symposium of Software Reliability Engineering (ISSRE), St. Malo, France. Citeseer (2004)

Del Grosso, C., Antoniol, G., Di Penta, M., Galinier, P., Merlo, E.: Improving network applications security: a new heuristic to generate stress testing data. In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation, pp. 1037–1043 (2005)

de Lima, D. F., Albuquerque, D., Dantas Filho, E., Perkusich, M., Perkusich, A.: Integrating reinforcement learning in software testing automation: a promising approach. In: Anais do III Workshop Brasileiro de Engenharia de Software Inteligente, pp. 39–41. SBC (2023)

Duchene, F., Groz, R., Rawat, S., Richier, J.-L.: Xss vulnerability detection using model inference assisted evolutionary fuzzing. In:2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pp. 815–817. IEEE (2012)

Duchene, F., Rawat, S., Richier, J.-L., Groz, R.: Kameleonfuzz: evolutionary fuzzing for black-box XSS detection. In: Proceedings of the 4th ACM Conference on Data and Application Security and Privacy, pp. 37–48 (2014)

Eberlein, M., Noller, Y., Vogel, T., Grunske, L.: Evolutionary grammar-based fuzzing. In: International Symposium on Search Based Software Engineering, pp. 105–120. Springer (2020)

Ebert, F., Castor, F., Serebrenik, A.: An exploratory study on exception handling bugs in java programs. J. Syst. Softw. 106 , 82–101 (2015)

Elyasov, A., Prasetya, I. S., Hage, J.: Search-based test data generation for Javascript functions that interact with the dom. In:2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE), pp. 88–99. IEEE (2018)

Esnaashari, M., Damia, A.H.: Automation of software test data generation using genetic algorithm and reinforcement learning. Expert Syst. Appl. 183 , 115446 (2021)

Fraser, G., Arcuri, A.: 1600 faults in 100 projects: automatically finding faults while achieving high coverage with EvoSuite. Empir. Softw. Eng. 20 (3), 611–639 (2015)

Fraser, G., Arcuri, A.: Evosuite: automatic test suite generation for object-oriented software. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, pp. 416–419 (2011)

Gan, J.-M., Ling, H.-Y., Leau, Y.-B.: A review on detection of cross-site scripting attacks (XSS) in web security. In: Advances in Cyber Security: Second International Conference, ACeS 2020, Penang, Malaysia, December 8–9, 2020, Revised Selected Papers 2, pp. 685–709. Springer (2021)

Gao, H., Feng, B., Zhu, L.: A kind of saaga hybrid meta-heuristic algorithm for the automatic test data generation. In: 2005 International Conference on Neural Networks and Brain, Vol. 1, pp. 111–114. IEEE (2005)

Del Grosso, C., Antoniol, G., Merlo, E., Galinier, P.: Detecting buffer overflow via automatic test input data generation. Comput. Oper. Res. 35 (10), 3125–3143 (2008)

Harman, M., Hu, L., Hierons, R. M., Baresel, A., Sthamer, H.: Improving evolutionary testing by flag removal. In: GECCO, pp. 1359–1366. Citeseer (2002)

Havrikov, N., Höschele, M., Galeotti, J. P., Zeller, A.: Xmlmate: Evolutionary xml test generation. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 719–722 (2014)

Htay, K. M., Othman, R. R., Amir, A., Zakaria, H. L., Ramli, N.: A pairwise t-way test suite generation strategy using gravitational search algorithm. In: 2021 International Conference on Artificial Intelligence and Computer Science Technology (ICAICST), pp. 7–12. IEEE (2021)

Hydara, I., Sultan, A.B.M., Zulzalil, H., Admodisastro, N.: Cross-site scripting detection based on an enhanced genetic algorithm. Indian J. Sci. Technol. 8 (30), 1–7 (2015)

Hydara, I., Sultan, A.B.M., Zulzalil, H., Admodisastro, N.: Current state of research on cross-site scripting (XSS)-a systematic literature review. Inf. Softw. Technol. 58 , 170–186 (2015)

Hydara, I., Sultan, A. B. M., Zulzalil, H., Admodisastro, N.: An approach for cross-site scripting detection and removal based on genetic algorithms. In: The Ninth International Conference on Software Engineering Advances ICSEA (2014)

Iannone, E., Di Nucci, D., Sabetta, A., De Lucia, A.: Toward automated exploit generation for known vulnerabilities in open-source libraries. In: 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC), pp. 396–400. IEEE (2021)

Imam, R., Anwer, F., Nadeem, M.: An effective and enhanced RSA based public key encryption scheme (XRSA). Int. J. Inf. Technol. 14 (5), 2645–2656 (2022)

Imam, R., Anwer, F.: An empirical study of secure and complex variants of RSA scheme. In: Cyber Security, Privacy and Networking, pp. 185–196. Springer (2022)

Imam, R., Areeb, Q. M., Alturki, A., Anwer, F.: Systematic and critical review of RSA based public key cryptographic schemes: past and present status. IEEE Access (2021)

Imam, R., Kumar, K., Raza, S. M., Sadaf, R., Anwer, F., Fatima, N., Nadeem, M., Abbas, M., Rahman, O.: A systematic literature review of attribute based encryption in health services. J. King Saud Univ.-Comput. Inf. Sci. (2022b)

Jan, S., Panichella, A., Arcuri, A., Briand, L.: Automatic generation of tests to exploit xml injection vulnerabilities in web applications. IEEE Trans. Softw. Eng. 45 (4), 335–362 (2017)

Jan, S., Panichella, A., Arcuri, A., Briand, L.: Search-based multi-vulnerability testing of xml injections in web applications. Empir. Softw. Eng. 24 (6), 3696–3729 (2019)

Jan, S., Nguyen, C. D., Arcuri, A., Briand, L.: A search-based testing approach for xml injection vulnerabilities in web applications. In: 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 356–366. IEEE (2017a)

Jawed, M. S., Sajid, M.: Xecryptoga: a metaheuristic algorithm-based block cipher to enhance the security goals. Evolving Systems, pp. 1–22 (2022)

Kayacik, H. G., Heywood, M., Zincir-Heywood, N.: On evolving buffer overflow attacks using genetic programming. In: Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pp. 1667–1674 (2006)

Kayacik, H. G., Zincir-Heywood, A. N., Heywood, M.: Evolving successful stack overflow attacks for vulnerability testing. In: 21st Annual Computer Security Applications Conference (ACSAC’05), p. 8. IEEE (2005)

Khanna, M., Chauhan, N., Sharma, D., Toofani, A., Chaudhary, A.: Search for prioritized test cases in multi-objective environment during web application testing. Arab. J. Sci. Eng. 43 (8), 4179–4201 (2018)

Khari, M., Sinha, A., Verdu, E., Crespo, R.G.: Performance analysis of six meta-heuristic algorithms over automated test suite generation for path coverage-based optimization. Soft. Comput. 24 (12), 9143–9160 (2020)

Khari, M., Vaishali, Kumar, M.: Search-based secure software testing: a survey. In: Software Engineering: Proceedings of CSI 2015, pp. 375–381. Springer (2019)

Khor, S., Grogono, P.: Using a genetic algorithm and formal concept analysis to generate branch coverage test data automatically. In: Proceedings 19th International Conference on Automated Software Engineering, 2004, pp. 346–349. IEEE (2004)

Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering (2007)

Kumar, A., Nadeem, M., Banka, H.: Nature inspired optimization algorithms: a comprehensive overview. Evol. Syst., pp. 1–16 (2022)

Lin, Y., Ong, Y. S., Sun, J., Fraser, G., Dong, J. S.: Graph-based seed object synthesis for search-based unit testing. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1068–1080 (2021)

Lin, Y., Sun, J., Fraser, G., Xiu, Z., Liu, T., Dong, J. S.: Recovering fitness gradients for interprocedural boolean flags in search-based testing. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 440–451 (2020)

Liu, G.-H., Wu, G., Tao, Z., Shuai, J.-M., Tang, Z.-C.: Vulnerability analysis for x86 executables using genetic algorithm and fuzzing. In: 2008 Third International Conference on Convergence and Hybrid Information Technology, vol. 2, pp. 491–497. IEEE (2008)

Liu, M., Li, K., Chen, T.: Security testing of web applications: a search-based approach for detecting SQL injection vulnerabilities. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 417–418 (2019)

Luo, Y.: Sqli-fuzzer: A SQL injection vulnerability discovery framework based on machine learning. In: 2021 IEEE 21st International Conference on Communication Technology (ICCT), pp. 846–851. IEEE (2021)

Lüdtke, S., Kraus, R., Barakat, R., Schneider, M. A.: Attack-based automation of security testing for IoT applications with genetic algorithms and fuzzing. In: 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 92–100. IEEE (2021)

Mann, M., Tomar, P., Sangwan, O.P.: Bio-inspired metaheuristics: evolving and prioritizing software test data. Appl. Intell. 48 (3), 687–702 (2018)

Mantere, T., Alander, J.T.: Evolutionary software engineering, a review. Appl. Soft Comput. 5 (3), 315–331 (2005)

Manès, V. J., Kim, S., Cha, S. K.: Ankou: guiding grey-box fuzzing towards combinatorial difference. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 1024–1036 (2020)

Mao, C.: Harmony search-based test data generation for branch coverage in software structural testing. Neural Comput. Appl. 25 (1), 199–216 (2014)

Mao, C., Wen, L., Chen, T. Y.: Adaptive random test case generation based on multi-objective evolutionary search. In: 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pp. 46–53. IEEE (2020)

Marashdeh, Z., Suwais, K., Alia, M.: A survey on SQL injection attack: Detection and challenges. In: 2021 International Conference on Information Technology (ICIT), pp. 957–962. IEEE (2021)

Marashdih, A. W., Zaaba, Z. F.: Detection and removing cross site scripting vulnerability in PHP web application. In:2017 International Conference on Promising Electronic Technologies (ICPET), pp. 26–31. IEEE (2017)

Marashdih, A. W., Zaaba, Z. F., Omer, H. K.: Web security: detection of cross site scripting in PHP web application using genetic algorithm. Int. J. Adv. Comput. Sci. Appl. (IJACSA) 8 (5) (2017)

Marculescu, B., Zhang, M., Arcuri, A.: On the faults found in rest APIs by automated test generation. ACM Trans. Softw. Eng. Methodol. 31 (3), 1–43 (2022)

McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verif. Reliab 14 (2), 105–156 (2004)

McMinn, P., Holcombe, M.: The state problem for evolutionary testing. In: Genetic and Evolutionary Computation Conference, pp. 2488–2498. Springer (2003)

McMinn, P., Shahbaz, M., Stevenson, M.: Search-based test input generation for string data types using the results of web queries. In: 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pp. 141–150. IEEE (2012)

Menéndez, H.D., Jahangirova, G., Sarro, F., Tonella, P., Clark, D.: Diversifying focused testing for unit testing. ACM Trans. Softw. Eng. Methodol. (TOSEM) 30 (4), 1–24 (2021)

Michael, C.C., McGraw, G., Schatz, M.A.: Generating software test data by evolution. IEEE Trans. Softw. Eng. 27 (12), 1085–1110 (2001)

Oster, N., Saglietti, F.: Automatic test data generation by multi-objective optimisation. In: International Conference on Computer Safety, Reliability, and Security, pp. 426–438. Springer (2006)

Padmanabhuni, B. M., Tan, H. B. K.: Light-weight rule-based test case generation for detecting buffer overflow vulnerabilities. In: 2015 IEEE/ACM 10th International Workshop on Automation of Software Test, pp. 48–52. IEEE (2015)

Paduraru, C., Melemciuc, M.-C., Stefanescu, A.: A distributed implementation using apache spark of a genetic algorithm applied to test data generation. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1857–1863 (2017)

Panichella, A., Kifetew, F.M., Tonella, P.: Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Trans. Software Eng. 44 (2), 122–158 (2017)

Panichella, A., Kifetew, F. M., Tonella, P.: Reformulating branch coverage as a many-objective optimization problem. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–10. IEEE (2015)

Pałka, D., Zachara, M., Wójcik, K.: Evolutionary scanner of web application vulnerabilities. In: International Conference on Computer Networks, pp. 384–396. Springer (2016)

Rauf, A., Anwar, S., Jaffer, M. A., Shahid, A. A.: Automated GUI test coverage analysis using GA. In: 2010 Seventh International Conference on Information Technology: New Generations, pp. 1057–1062. IEEE (2010)

Rawat, S., Ceara, D., Mounier, L., Potet, M.-L.: Combining static and dynamic analysis for vulnerability detection. arXiv preprint https://arxiv.org/abs/1305.3883 (2013)

Rawat, S., Mounier, L.: An evolutionary computing approach for hunting buffer overflow vulnerabilities: a case of aiming in dim light. In: 2010 European Conference on Computer Network Defense, pp. 37–45. IEEE (2010)

Ren, T., Wang, X., Li, Q., Wang, C., Dong, J., Guo, G.: Vulnerability mining technology based on genetic algorithm and model constraint. In: IOP Conference Series: Materials Science and Engineering, Vol. 750, p. 012168. IOP Publishing (2020)

Reungsinkonkarn, A., Apirukvorapinit, P.: Bug detection using particle swarm optimization with search space reduction. In: 2015 6th International Conference on Intelligent Systems, Modelling and Simulation, pp. 53–57. IEEE (2015)

Rodrigues, D.S., Delamaro, M.E., Corrêa, C.G., Nunes, F.L.: Using genetic algorithms in test data generation: a critical systematic mapping. ACM Comput. Surv. 51 (2), 1–23 (2018)

Romano, D., Di Penta, M., Antoniol, G.: An approach for search based testing of null pointer exceptions. In: 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation, pp. 160–169. IEEE (2011)

Saber, T., Delavernhe, F., Papadakis, M., O’Neill, M., Ventresque, A.: A hybrid algorithm for multi-objective test case selection. In: 2018 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8. IEEE (2018)

Seesing, A., Gross, H.-G.: A genetic programming approach to automated test generation for object-oriented software. Int. Trans. Syst. Sci. Appl. 1 (2) (2006)

Shahbazi, A., Miller, J.: Black-box string test case generation through a multi-objective optimization. IEEE Trans. Softw. Eng. 42 (4), 361–378 (2015)

Shuai, B., Li, H., Zhang, L., Zhang, Q., Tang, C.: Software vulnerability detection based on code coverage and test cost. In: 2015 11th International Conference on Computational Intelligence and Security (CIS), pp. 317–321. IEEE (2015a)

Shuai, B., Li, M., Li, H., Zhang, Q.: Test case generation for vulnerability detection using genetic algorithm. In: 4rd Int. Conf. Consumer Electronics, Communications and Networks, pp. 1198–1203 (2015)

Shuai, B., Li, M., Li, H., Zhang, Q., Tang, C.: Software vulnerability detection using genetic algorithm and dynamic taint analysis. In: 2013 3rd International Conference on Consumer Electronics, Communications and Networks, pp. 589–593. IEEE (2013)

Silva, R.A., de Souza, S. R. S., de Souza, P. S. L.: A systematic review on search based mutation testing. Inf. Softw. Technol. 81 , 19–35 (2017)

Skaruz, J., Seredynski, F.: Detecting web application attacks with use of gene expression programming. In: 2009 IEEE Congress on Evolutionary Computation, pp. 2029–2035. IEEE (2009)

Soltani, M., Derakhshanfar, P., Devroey, X., Van Deursen, A.: A benchmark-based evaluation of search-based crash reproduction. Empir. Softw. Eng. 25 , 96–138 (2020)

Sparks, S., Embleton, S., Cunningham, R., Zou, C.: Automated vulnerability analysis: leveraging control flow for evolutionary input crafting. In: Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007), pp. 477–486. IEEE (2007)

Stallenberg, D. M., Panichella, A.: Jcomix: A search-based tool to detect xml injection vulnerabilities in web applications. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1090–1094 (2019)

Thomé, J., Shar, L.K., Bianculli, D., Briand, L.: An integrated approach for effective injection vulnerability analysis of web applications through security slicing and hybrid constraint solving. IEEE Trans. Software Eng. 46 (2), 163–195 (2018)

Thomé, J., Gorla, A., Zeller, A.: Search-based security testing of web applications. In: Proceedings of the 7th International Workshop on Search-Based Software Testing, pp. 5–14 (2014)

Thomé, J., Shar, L. K., Bianculli, D., Briand, L.: Search-driven string constraint solving for vulnerability detection. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pp. 198–208. IEEE (2017)

Tlili, M., Wappler, S., Sthamer, H.: Improving evolutionary real-time testing. In: Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pp. 1917–1924 (2006)

Tonella, P.: Evolutionary testing of classes. ACM SIGSOFT Softw. Eng. Notes 29 (4), 119–128 (2004)

Umar, K., Sultan, A. B., Zulzalil, H., Admodisastro, N., Abdullah, M. T.: Prevention of attack on Islamic websites by fixing SQL injection vulnerabilities using co-evolutionary search approach. In: The 5th International Conference on Information and Communication Technology for The Muslim World (ICT4M), pp. 1–6. IEEE (2014)

Umar, K., Sultan, A. B., Zulzalil, H., Admodisastro, N., Abdullah, M. T.: Formulation of SQL injection vulnerability detection as grammar reachability problem. In: 2018 International Conference on Information and Communication Technology for the Muslim World (ICT4M), pp. 179–184. IEEE (2018)

Vulnerability distribution of cve security vulnerabilities by types

Wang, W., Guo, X., Li, Z., Zhao, R.: Test case generation based on client-server of web applications by memetic algorithm. In: 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE), pp. 206–216. IEEE (2019a)

Wang, W., Wu, S., Li, Z., Zhao, R.: Parallel evolutionary test case generation for web applications. Inf. Softw. Technol. 155 , 107113 (2023)

Wang, Y., Wang, Y.: Use neural network to improve fault injection testing. In: 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 377–384. IEEE (2017)

Wang, Y., Wu, Z., Wei, Q., Wang, Q.: Field-aware evolutionary fuzzing based on input specifications and vulnerability metrics. In: 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS), pp. 1–7. IEEE (2019b)

Wappler, S., Lammermann, F.: Using evolutionary algorithms for the unit testing of object-oriented software. In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation, pp. 1053–1060, (2005)

Wegener, J., Baresel, A., Sthamer, H.: Evolutionary test environment for automatic structural testing. Inf. Softw. Technol. 43 (14), 841–854 (2001)

Wegener, J., Buhr, K., Pohlheim, H.: Automatic test data generation for structural testing of embedded software systems by evolutionary testing. In: Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation, pp. 1233–1240 (2002)

Wei, Q., Li, Y., Zhang, Y.: A new method of evolutionary testing for path coverage. In: 2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 79–86. IEEE (2018)

Wu, Z., Atwood, J. W., Zhu, X.: A new fuzzing technique for software vulnerability mining. In: International Conference on Software Engineering. Citeseer (2009)

Xu, X., Jiao, L., Zhu, Z.: Boosting search based software testing by using ensemble methods. In: 2018 IEEE Congress on Evolutionary Computation (CEC), pp. 1–10. IEEE (2018)

Yao, X., Gong, D., Li, B., Dang, X., Zhang, G.: Testing method for software with randomness using genetic algorithm. IEEE Access 8 , 61999–62010 (2020)

Ye, J., Feng, C., Tang, C.: A fuzzer based on a fine-grained deeper strategy. In: 2017 4th International Conference on Information Science and Control Engineering (ICISCE), pp. 24–28. IEEE (2017)

Zhu, X. Y., Wu, Z. Y.: A new fuzzing technique using niche genetic algorithm. In: Advanced Materials Research, volume 756, pp. 4050–4058. Trans Tech Publ (2013)

Zhu, Z., Jiao, L., Xu, X.: Combining search-based testing and dynamic symbolic execution by evolvability metric. In: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 59–68. IEEE (2018)

Download references

Author information

Authors and affiliations.

Department of Computer Science, Aligarh Muslim University, Aligarh, UP, 202002, India

Fatma Ahsan & Faisal Anwer

You can also search for this author in PubMed   Google Scholar

Contributions

All the authors are contributed equally.

Corresponding author

Correspondence to Fatma Ahsan .

Ethics declarations

Conflict of interest.

There is no Conflict of interest and no data available for this review paper.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Search string, selected primary studies, venue details and list of abbreviations, and quality assessment

See Tables 9 , 10 and 11 .

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Ahsan, F., Anwer, F. A systematic literature review on software security testing using metaheuristics. Autom Softw Eng 31 , 44 (2024). https://doi.org/10.1007/s10515-024-00433-0

Download citation

Received : 10 August 2023

Accepted : 13 March 2024

Published : 23 May 2024

DOI : https://doi.org/10.1007/s10515-024-00433-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Meta-heuristic
• Optimization algorithm
• Software security testing
• Code coverage
• Program crash

Advertisement

• Find a journal
• Publish with us
• Track your research

Salesforce is closed for new business in your area.