data envelopment analysis literature review

Data Envelopment Analysis and Big Data: A Systematic Literature Review with Bibliometric Analysis

• First Online: 17 December 2021

Cite this chapter

• Vincent Charles 6 ,
• Tatiana Gherman 7 &
• Joe Zhu 8  

Part of the book series: International Series in Operations Research & Management Science ((ISOR,volume 312))

927 Accesses

7 Citations

Data envelopment analysis (DEA) is a powerful data-enabled, big data science tool for performance measurement and management, which over time has been applied across a myriad of domains. Over the past years, various advancements in big data have captured the attention of DEA scholars, which in turn, has translated into the emergence of new research strands. In the present work, we perform a systematic literature review with bibliometric analysis of studies integrating DEA with big data, in an attempt to answer the question: what are the current avenues of research for such studies? The results obtained are further complemented with a thematic analysis. Among others, findings indicate that big data is still a new entrant within the DEA literature, that most of the studies have focused on developing faster and more accurate computational techniques to handle problems with a large number of decision-making units (DMUs), and that most of the studies have been carried out in the area of environmental efficiency evaluation. This work should contribute to the construction of an overview of the existing literature on DEA-big data studies, as well as stimulate the interest in the topic.

• Data envelopment analysis
• Data-enabled analytics
• Systematic literature review
• Bibliometric analysis

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

An, Q., Wen, Y., Xiong, B., Yang, M., & Chen, X. (2017). Allocation of carbon dioxide emission permits with the minimum cost for Chinese provinces in big data environment. Journal of Cleaner Production, 142 , 886–893.

Article   Google Scholar  

Badiezadeh, T., Saen, R. F., & Samavati, T. (2018). Assessing sustainability of supply chains by double frontier network DEA: A big data approach. Computers and Operations Research, 98 , 284–290.

Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and scale inefficiencies in Data Envelopment Analysis. Management Science, 30 , 1078–1092.

Bizer, C., Boncz, P., Brodie, M. L., & Erling, O. (2012). The meaningful use of big data: Four perspectives-four challenges. ACM SIGMOD Record, 40 (4), 56–60.

Brynjolfsson, E., Hitt, L. M., & Kim, H. H. (2011). Strength in numbers: How does data-driven decision making affect firm performance? Social Science Electronic Publishing.

Google Scholar  

Chang, Y. T., Zhang, N., Danao, D., & Zhang, N. (2013). Environmental efficiency analysis of transportation system in China: A non-radial DEA approach. Energy Policy, 58 , 277–283.

Charles, V., & Gherman, T. (2013). Achieving competitive advantage through big data. Strategic implications. Middle-East Journal of Scientific Research, 16 (8), 1069–1074.

Charles, V., & Gherman, T. (2018). Big data and ethnography: Together for the greater good. In A. Emrouznejad & V. Charles (Eds.), Big data for the greater good (pp. 19–34). Springer.

Charles, V., Tavana, M., & Gherman, T. (2015). The right to be forgotten – Is privacy sold out in the big data age? International Journal of Society Systems Science, 7(4), 283-298.

Charles, V., Tsolas, I. E., & Gherman, T. (2018). Satisficing data envelopment analysis: A Bayesian approach for peer mining in the banking sector. Annals of Operations Research, 269 (1–2), 81–102.

Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2 (6), 429–444.

Chen, C., Achtari, G., Majkut, K., & Sheu, J.-B. (2017). Balancing equity and cost in rural transportation management with multi-objective utility analysis and data envelopment analysis: A case of Quinte West. Transportation Research Part A, 95 , 148–165.

Chen, C. P., & Zhang, C. Y. (2014). Data-intensive applications, challenges, techniques and technologies: A survey on Big Data. Information Sciences, 275 , 314–347.

Chen, L., & Jia, G. (2017). Environmental efficiency analysis of China’s regional industry: A data envelopment analysis (DEA) based approach. Journal of Cleaner Production, 142 , 846–853.

Chen, W. C., & Cho, W. J. (2009). A procedure for large-scale DEA computations. Computers & Operations Research, 36 (6), 1813–1824.

Chen, W. C., & Lai, S. Y. (2015). Determining radial efficiency with a large data set by solving small-size linear programs. Annals of Operations Research, 250 , 147–166.

Chu, J.-F., Wu, J., & Song, M.-L. (2018). An SBM-DEA model with parallel computing design for environmental efficiency evaluation in the big data context: A transportation system application. Annals of Operations Research, 270 (1-2), 105–124.

Cook, W. D., Tone, K., & Zhu, J. (2014). Data envelopment analysis: Prior to choosing a model. Omega, 44 , 1–4.

Dulá, J. H. (2011). An algorithm for data envelopment analysis. INFORMS Journal on Computing, 23 (2), 284–296.

Dulá, J. H., & López, F. J. (2009). Preprocessing DEA. Computers & Operations Research, 36 (4), 1204–1220.

Fan, M.-W., Ao, C.-C., & Wang, X.-R. (2019). Comprehensive method of natural gas pipeline efficiency evaluation based on energy and big data analysis. Energy, 188 , 116069.

Farrell, M. J. (1957). The measurement of productive efficiency. Journal of the Royal Statistical Society: Series A, 120 (3), 253–281.

Gong, B., Guo, D., Zhang, X., & Cheng, J. (2017). An approach for evaluating cleaner production performance in iron and steel enterprises involving competitive relationships. Journal of Cleaner Production, 142 , 739–748.

He, Z., He, Y., Liu, F., & Zhao, Y. (2019). Big data-oriented product infant failure intelligent root cause identification using Associated tree and fuzzy DEA. IEEE Access, 7 (8667817), 34687–34698.

Herranz, R. E., Estévez, P. G., Oliva, M. A. D. V. Y., & Dé, R. (2017). Leveraging financial management performance of the Spanish aerospace manufacturing value chain. Journal of Business Economics and Management, 18 (5), 1005–1022.

Khezrimotlagh, D., Zhu, J., Cook, W. D., & Toloo, M. (2019). Data envelopment analysis and big data. European Journal of Operational Research, 274 (3), 1047–1054.

Kiani Mavi, R., & Kiani Mavi, N. (2021). National eco-innovation analysis with big data: A common-weights model for dynamic DEA. Technological Forecasting and Social Change, 162 , 120369.

Kiani Mavi, R., Saen, R. F., & Goh, M. (2019). Joint analysis of eco-efficiency and eco-innovation with common weights in two-stage network DEA: A big data approach. Technological Forecasting and Social Change, 144 , 553–562.

Laney, D. (2001). 3D data management: Controlling data volume, velocity and variety. Applications delivery strategies . META Group (now Gartner) [online] http://blogs.gartner.com/doug-laney/files/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf .

Li, L., Hao, T., & Chi, T. (2017). Evaluation on China’s forestry resources efficiency based on big data. Journal of Cleaner Production, 142 , 513–523.

Liu, X., Chu, J., Yin, P., & Sun, J. (2017). DEA cross-efficiency evaluation considering undesirable output and ranking priority: A case study of eco-efficiency analysis of coal-fired power plants. Journal of Cleaner Production, 142 , 877–885.

Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., & Hung Byers, A. (2011). Big data: The next frontier for innovation, competition and productivity. McKinsey Quarterly . Retrieved on 13 January 2021 from https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Digital/Our%20Insights/Big%20data%20The%20next%20frontier%20for%20innovation/MGI_big_data_exec_summary.pdf .

Michael, K., & Miller, K. W. (2013). Big data: New opportunities and new challenges [guest editors’ introduction]. Computer, 46 (6), 22–24.

Mubin, O., Arsalan, M., & Al Mahmud, A. (2018). Tracking the follow-up of work in progress papers. Scientometrics, 114 , 1159–1174.

Müller, O., Fay, M., & Vom Brocke, J. (2018). The effect of big data and analytics on firm performance: An econometric analysis considering industry characteristics. Journal of Management Information Systems, 35 (2), 488–509.

Provost, F., & Fawcett, T. (2013). Data science for business: What you need to know about data mining and data-analytic thinking . O’Reilly.

Song, M.-L., Fisher, R., Wang, J.-L., & Cui, L.-B. (2018). Environmental performance evaluation with big data: Theories and methods. Annals of Operations Research, 270 (1–2), 459–472.

Song, M., Du, Q., & Zhu, Q. (2017). A theoretical method of environmental performance evaluation in the context of big data. Production Planning and Control, 28 (11–12), 976–984.

Taboada, G. L., & Han, L. (2020). Exploratory data analysis and data envelopment analysis of urban rail transit. Electronics, 9 (8), 1–29.

Tayal, A., Solanki, A., & Singh, S. P. (2020). Integrated frame work for identifying sustainable manufacturing layouts based on big data, machine learning, meta-heuristic and data envelopment analysis. Sustainable Cities and Society, 62 , 102383.

Wu, X., Zhu, X., Wu, G. Q., & Ding, W. (2014). Data mining with big data. IEEE Transactions on Knowledge and Data Engineering, 26 (1), 97–107.

Zelenyuk, V. (2020). Aggregation of inputs and outputs prior to Data Envelopment Analysis under big data. European Journal of Operational Research, 282 (1), 172–187.

Zhan, J., Zhang, F., Li, Z., Zhang, Y., & Qi, W. (2020). Evaluation of food security based on DEA method: A case study of Heihe River Basin. Annals of Operations Research, 290 (1–2), 697–706.

Zhang, Y., Huang, Y., Porter, A. L., Zhang, G., & Lu, J. (2019). Discovering and forecasting interactions in big data research: A learning-enhanced bibliometric study. Technological Forecasting and Social Change, 146 , 795–807.

Zhu, J. (2020). DEA under big data: Data enabled analytics and network data envelopment analysis. Annals of Operations Research , 1–23.

Zhu, Q., Li, X., Li, F., & Zhou, D. (2020). The potential for energy saving and carbon emission reduction in China’s regional industrial sectors. Science of the Total Environment, 716 , 135009.

Zhu, Q., Wu, J., Li, X., & Xiong, B. (2017). China’s regional natural resource allocation and utilization: A DEA-based approach in a big data environment. Journal of Cleaner Production, 142 , 809–818.

Zhu, Q., Wu, J., & Song, M. (2018). Efficiency evaluation based on data envelopment analysis in the big data context. Computers and Operations Research, 98 , 291–300.

Download references

Acknowledgement

The authors are thankful to the reviewers for their valuable feedback on the previous version of this research.

Author information

Authors and affiliations.

University of Wales Trinity Saint David, Birmingham, UK

Vincent Charles

Faculty of Business and Law, University of Northampton, Northampton, UK

Tatiana Gherman

Business School, Worcester Polytechnic Institute, Worcester, MA, USA

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Vincent Charles .

Editor information

Editors and affiliations.

Foisie Business School, Worcester Polytechnic Institute, Worcester, MA, USA

Buckingham Business School, University of Buckingham, Birmingham, UK

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Charles, V., Gherman, T., Zhu, J. (2021). Data Envelopment Analysis and Big Data: A Systematic Literature Review with Bibliometric Analysis. In: Zhu, J., Charles, V. (eds) Data-Enabled Analytics. International Series in Operations Research & Management Science, vol 312. Springer, Cham. https://doi.org/10.1007/978-3-030-75162-3_1

Download citation

DOI : https://doi.org/10.1007/978-3-030-75162-3_1

Published : 17 December 2021

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-75161-6

Online ISBN : 978-3-030-75162-3

eBook Packages : Business and Management Business and Management (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Search Menu
• Advance articles
• Special issues
• Virtual issues
• Author Guidelines
• Submission Site
• Open Access
• Reasons to submit
• About IMA Journal of Management Mathematics
• About the Institute of Mathematics and its Applications
• Editorial Board
• Area Editors
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

• 1. Introduction
• 2. Data envelopment analysis
• 3. Malmquist–Luenberger productivity index
• 4. Structured literature review method
• 5. Findings from the literature review: evolution of the field and analysis of the historiograph
• 6. Recommendation for energy policy makers
• 7. Conclusions and direction for future research
• Acknowledgements
• Data statement
• < Previous

Eco-efficiency considering NetZero and data envelopment analysis: a critical literature review

• Article contents
• Figures & tables
• Supplementary Data

Ali Emrouznejad, Marianna Marra, Guo-liang Yang, Maria Michali, Eco-efficiency considering NetZero and data envelopment analysis: a critical literature review, IMA Journal of Management Mathematics , Volume 34, Issue 4, October 2023, Pages 599–632, https://doi.org/10.1093/imaman/dpad002

• Permissions Icon Permissions

We highlight the state of the art in the eco-efficiency measurement using data envelopment analysis, including Malmquist–Luenberger productivity index. We also consider productivity change over time, provide directions for future studies in the field and gather the most recent policy suggestions for governments, organizations and sectors for reducing CO 2 emissions. A structured literature search of the Web of Science academic database reveals 311 papers published between 1989 and 2022. We carry out network analysis of citations to show the evolution of the literature in this research topic. In doing so, we (a) examine the key-route main path of knowledge flows, (b) provide basic bibliometric information about the most active journals and authors, (c) conduct a qualitative in-depth analysis of the identified most important studies and (d) identify the research fronts and relate them to the emerging issues on the topic researched, focusing on the most recent period between 2000 and 2022. Based on the insights of the literature review, the second part of this paper critically analyses the papers on the key-route (main path) of this subject. This review can be used as guidance and a starting point for researchers and practitioners who want to further investigate optimal policies to reach NetZero.

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1471-6798
• Copyright © 2024 Institute of Mathematics and its Applications
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.8(12); 2022 Dec

A data-envelopment analysis-based systematic review of the literature on innovation performance

Associated data.

Data included in article/supp. material/referenced in article.

Innovation imbued in every sector in every part of the world is essential to growth and development. The National Innovation Systems (NIS) use their resources to support economies in different countries foster a novel environment. Therefore, this study is an attempt to examine the efficiency of NIS as measured by scholars in the past using Data Envelopment Analysis (DEA). Through a systematic literature review, this study aims to show the current dearth of studies on the efficiency of NIS. The literature is categorized accordingly and provides a framework with recommendations for future research. With the advent of technical tools, DEA programming methods other than traditional DEA such as network, relational network, dynamic network, slack based model, and super efficiency DEA have emerged. This enables the calculation of innovation efficiency to be efficient and reliable. As a result, DEA is regarded as a powerful tool for assessing the relative efficiency of NIS, which employs multiple inputs to generate multiple outputs. The results also suggest that future research is needed on the efficiency of innovation by paying more attention to cross-countries studies based on regions, geographical areas, participation in free trade blocs, and a group of tie-up countries involved, especially with comparative analyses regardless of the country classification. Another important observation is that there are few studies that focus on the efficiency of middle- and low-income countries. The comparative analysis of innovation efficiency across income groups enables NIS to benchmark itself against best-in-class innovators and improve their innovation performance and ranking. These findings provide an opportunity to further investigate how NIS add value and sustainability to countries by improving resource management capabilities to improve innovation efficiency.

Data envelopment analysis; DEA; Efficiency; National innovation system; NIS; Technical efficiency

1. Introduction

The rapidly growing trend of globalization requires that each country creates unique competitive advantages to remain internationally competitive. Accordingly, countries around the world continue to plan and implement a variety of innovative measures to either maintain or create competitive advantages. Innovation is one of the key elements to improve competitiveness in the market ( Dereli, 2015 ; Ivanova and Cepel, 2018 ; Rajapathirana and Hui, 2018 ). The introduction of innovation by an organization encourages among others to find new methods to carve out a unique niche. However, the innovation process requires the support of various aspects. For example, technology is currently one of the most important features for advancing the frontiers of innovations ( Dodgson et al., 2006 ; Koellinger, 2008 ). Similarly, the need for appropriate conditions to facilitate these developments is paramount; including having suitable market-friendly business models and processes ( Bouwman et al., 2019 ).

We start with ideas presented in the very first papers that used the innovation system concept, namely Freeman (1982) and Lundvall (1985) . Freeman's analysis focuses on macro phenomena, while Lundvall (1985) focuses on micro phenomena. The theory of National Innovation Systems or NIS has been widely accepted since the mid-1980s, as the most competitive countries in the world have extensive and highly developed national innovation systems ( Freeman, 1987 ; Lundvall, 2007 ). The concept of NIS was developed solely as a networking medium between both the private and public sectors to help initiate, modify, import, and even disseminate the new technology-based innovations brought to the market ( Freeman, 1987 ). NIS has been defined as a set of linkages facilitated by the actors involved in innovation, creating an interactive network between them ( Gokhberg and Roud, 2016 ; Lala and Sinha, 2019 ). The network was built to provide an efficient background for innovation development while addressing regulatory requirements.

The main motive for establishing an agency similar to NIS is to develop and promote innovation practices, focus on research and development, and enable economic development by deploying these cutting-edge innovations through entrepreneurial ventures. Dahlman (1994) mentions in the definition of NIS, that it is a network of policies and an institution that enable the introduction of new technologies in the market. As innovation plays an important role in shaping global economic development in creating and maintaining competitiveness, researchers are increasingly interested in examining the concept of innovation from a global perspective. Since the mid-1980s, the NIS has been used as the primary analytical target in the study of innovation processes ( Diez and Kiese, 2009 ). Since the late 1990s, researchers have used the NIS concept in their study ( Edler and Fagerberg, 2017 ; Guan and Chen, 2012 ; Kuhlmann and Ordóñez-Matamoros, 2017 ; Liu et al., 2015 ; Watkins et al., 2015 ) and this concept can be helpful as a generalized conceptual framework for examining differences in efficiency across countries ( Teixeira, 2014 ).

The importance of NIS is quite high nowadays, especially given the increasing number of innovations proposed by various market players ( Golichenko, 2016 ; Lundvall, 2016 ). Recent changes in the global economy over time have led to new perspectives in measuring innovation performance. Communication and collaboration among stakeholders have driven the development of innovations and explained the growth patterns of nonlinear innovations ( Todttling and Trippl, 2005 ). This development has led to a shift in perspective in measuring innovation performance from a single input dimension to multiple input dimensions ( Pan et al., 2010 ). Since the innovation process is quite complex, its performance should be measured taking into account multiple dimensions rather than based on a single unit of input and output ( Tidd and Bessant, 2020 ).

The application and diffusion of NIS among the concerned actors in an economy are done through various non-parametric methods. This method essentially aims to measure the relative efficiency of a group of decision-making units (DMUs). Data Envelopment Analysis or DEA is one such technical product based on linear programming that can convert the inputs and outputs used into a single measure of performance ( Kong et al., 2021 ; Xiong et al., 2020 ). This technique based on frontier transformation has the potential to measure efficiency of different countries. It is a non-parametric method that does not require the inclusion of predetermined work processes to arrive at a decision state ( Afzal, 2014 ; Cooper et al., 2006 ). The DEA method has been used extensively in various cross-border studies to measure the efficiency of NIS ( Nasierowski and Arcelus, 2003 ). This approach was developed by Farrell (1957) and Charnes et al. (1978) . DEA is a mathematical programming method for determining efficiency levels using data to measure efficiency values when forming efficiency frontier patterns ( Kotsemir, 2013 ). Jaloudi (2019) , and Nasierowski and Arcelus (2012) state that DEA is used to convert the minimum number of input units to the maximum number of outputs when measuring the efficiency of the required sample.

Technical efficiency (TE) is a measure of how well a DMU succeeds in obtaining the maximum output from a given input. TE measures output relative to the output of an efficient isoquant curve. Efficient firms produce at the production frontier or in efficiently equal quantities. TE can be calculated from the ratio of the sum of weighted outputs to the sum of weighted inputs ( Cooper et al., 2006 ), as follows:

where ‘ x ’ and ‘ y ’ are inputs and outputs, ‘ v ’ and ‘ u ’ are input and output weights, respectively, ‘ q ’ is the number of inputs ( q = 1 , 2 , . . . , Q ); ‘ p ’ is the number of outputs ( p = 1 , 2 , . . , P ); and ‘ j ’ represents j th DMU.

The use of DEA in the NIS is the most common in the literature to date, given its flexibility compared to parametric approaches, both in practice and in theory ( Kou et al., 2016 )Several empirical studies have used the DEA approach to assess the efficiency of NIS ( Cai, 2011 ; Carayannis et al., 2016 , Carayannis et al., 2015 ; Liu et al., 2017 ; Xu and Cheng, 2013 ) of NIS ( Alnafrah, 2021 ; Bresciani et al., 2021 ; Kotsemir, 2013 ; Shin et al., 2018 ; Zeng et al., 2021 ; Zhang, 2013 ). Since both the DEA and NIS concepts are proving to be highly relevant and effective in today's world, this study attempts to merge the two concepts to determine the technical efficiency of NIS through the application of DEA. The main objective of this systematic review is to address the lack of literature regarding the efficiency of NIS based on the application of DEA. The research questions of the study are as follows:

• 1. How many previous studies have examined innovation systems at the national and regional levels?
• 2. What methods do researchers commonly use to assess the efficiency of innovation systems?
• 3. What country taxonomy is the focus of studies on innovation systems efficiency?
• 4. What DEA model have researchers used to assess innovation systems efficiency?
• 5. What types of measurements have the researchers used?
• 6. What income groups do the researchers consider when assessing innovation systems efficiency?

The study would be essentially entail the following steps ( Salim et al., 2019 ):

• 1. In the first phase, the research articles that contain the application of NIS, DEA and other relevant information sought in the objectives of the study would be shortlisted.
• 2. In the second phase, the short-listed articles would be classified and coded according to their specific characteristics and reviewed.
• 3. In the third phase, summary presentation the afore-mentioned review.
• 4. Finally, propose a framework for address gap of literature.

2. Research methodology

The study attempts to use a systematic literature review approach as this is an effective method ( Levy and Ellis, 2006 ; Snyder, 2019 ) to capture the results of a large amount of stored information ( Popay et al., 2006 ). Several researchers in the past have used the systematic review approach as a transparent filtering method for the literature reviewed in an unbiased approach ( Centobelli et al., 2017 ; Tranfield et al., 2003 ). It allows researchers to delve deeper into a topic and gain better insights. One of the advantages of a systematic literature review is that it allows researchers to publish the content of the literature review in question in relation to specific emerging issues while ensuring a smooth flow of information. According to Godinho and Veloso (2013) , the process of classifying the literature requires the following six steps:

• • 1: Research of the relevant literature on the current state of knowledge;
• • 2: A classification process using a structured code that includes logical reasoning;
• • 3: Classification of the reviewed papers based on the classification process;
• • 4: The results of the study are presented;
• • 5: A detailed analysis of the existing gaps; and
• • 6: Suggestions for future research.

Most of the articles used in this literature review were obtained from Google Scholar, Web of Science, and Scopus databases given the large database, high quality and authenticity of sources. The database was searched using keywords such as National Innovation Systems, Data Envelopment Analysis, NIS, DEA, efficiency and innovation systems, and for the period between 2010 and 2021. The total number of articles found was 387. With the gradual exclusion of studies based on applied criteria ( Liberati et al., 2009 ), only the articles dealing with NIS and the DEA approach were selected. The incomplete articles on this topic were excluded, leading to only 74 short-listed articles for review and data synthesis. Fig. 1 above illustrates the process of selecting articles for the review.

Records selection process.

2.1. Classification and coding

As mentioned earlier, the coding procedure used in this systematic literature review to classify articles and assign specific codes is detailed in Table 1 .

Classification and coding.

3. Results of the literature review

Table 2 provides information on the authors, the journals in which the articles were published, and the authors' origin.

Listed of selected articles.

Table 3 shows the code classification of the articles based on the six parameters identified for review.

Detailed review.

4. Descriptive analysis

The analysis of NIS can be divided into two approaches ( Belitz et al., 2011 ): a descriptive approach based on case studies ( Nelson, 1993 ) and a theoretical approach based on secondary research and quantitative indicators ( Lundvall, 2010 ). Fig. 2 shows the number of articles published per year. It was found that 13 articles were published in 2020, followed by 10 articles in 2019 and 2021 respectively, eight articles in 2016, and seven articles in 2014. Subsequently, five articles were published in 2018, follow by four articles in 2011, 2013, and 2017, and the fewest, three articles were published in 2010 and 2015. In comparison to the global development of innovation, the number of articles published on innovation performance is comparatively modest.

Number of articles published each year.

4.1. Context of determinants

An innovation system is a multi-level concept ( Carayannis et al., 2016 ) in which national, regional, and sectoral innovation systems ( Schrempf et al., 2013 ; Edquist, 2009 ; Archibugi, 1996 ) can coexist and develop together in the same country. Fig. 3 shows the overall distribution of articles in the national and regional context. As can be seen, the highest number of articles in the overview is actually the national context, up to 57%. The remaining 43% of the articles deal with a specific geographic region, which has been linked to the regional context in the studies. Nevertheless, more attention is paid to the national context ( Kou et al., 2016 ) than to the regional context. Therefore, strengthening regional innovation efficiency is necessary to bridge the innovation-based economic gap between heterogeneous regions and improve their innovation efficiency at the national level ( OECD, 2008 ).

Distribution of determinants contexts.

4.2. Methodological context

According to Jabbour (2013) and Amui et al. (2017) , the methodological approach of each work is analyzed according to the following classifications: qualitative, quantitative, conceptual and/or, empirical, case studies and/or interviews and surveys. Fig. 4 shows the frequency of the methodological contexts used in this study. It is undeniable that all of the articles reviewed use a quantitative approach to measure the efficiency of innovation. Data sources could include either primary or secondary data with the methods used to collect them, including empirical data collected through survey or case study methods.

Distribution of research approaches.

4.3. National contexts

The categorization of the world's various economies into developed, developing and least developed countries (LDCs) has allowed researchers to specify the evidence required for each type of these countries. With the development of the innovation system concept, tools for evaluating and measuring such systems were developed in different countries ( Lacka and Brzezicki, 2021 ; Varblane et al., 2007 ). Fig. 5 shows the frequency of this study carried out according to the state of the national context. Studies on the innovation efficiency of developing countries are comparatively higher, reaching up to 89%, than those on developed countries (64%). It also demonstrates that studies on innovation efficiency involving the least developed countries are limited, signaling the need for more in-depth analyses of innovation efficiency in the LDCs to facilitate these countries develop good innovation policies.

Distribution of national contexts.

4.4. DEA models

Most efficiency studies are motivated by the desire to estimate economic performance based on parametric or non-parametric methods ( Ajibefun, 2008 ; Asmare and Begashaw, 2018 ; Murillo-Zamorano and Vega-Cervera, 2001 ). The majority of the scholars evaluate innovation efficiency using DEA as a non-parametric approach. Fig. 6 shows the distribution of the DEA models. The traditional DEA model (39 studies) is used the most in measuring the efficiency of innovation, followed by the N-DEA model (16 studies). The B-DEA and RN-DEA models were used six times each, while the SMB-DEA and SE-DEA models were used in four different studies to assess innovation efficiency. Meanwhile, the DN-DEA model (2 studies) is the least used model in measuring innovation efficiency. As a result, in addition to T-DEA, evolved DEA models should be used to quantify innovation efficiency because they can provide reliable and scientifically established measurement values.

Distribution of DEA models.

4.5. Result areas

The NIS input and output variables can be quantified to assess a country's innovation efficiency and productivity ( Banker et al., 2013 ; Lacka and Brzezicki, 2021 ; Grilo and Santos, 2015 ). Fig. 7 shows the distribution of the result areas. Around 80% of articles used DEA to measure efficiency, while 11% of articles used Malmquist approach to measure productivity. In the meantime, only 9% of articles study both the efficiency and productivity of innovation. Also observed, regional innovation which assesses the efficiency and productivity of innovation is considerably low compared to national innovation. As a result, adequate attention should be paid to measuring efficiency and productivity at the regional level in order to improve regional innovation performance, which in turn will help to improve national innovation performance and ranking.

Distribution of result areas.

4.6. Income groups

The World Bank has divided countries around the world into high, upper-middle, lower-middle, and low-income countries based on their gross domestic product values. Choi and Zo (2019) and Leontitsis et al. (2018) assess the efficiency of NIS by classifying them by income group. Fig. 8 shows the distribution of the countries examined by income group. According to the review, 48% of the studies involved upper-middle income countries, while 33% involved high-income countries. However, the observation revealed that only 15% of low-middle income countries and 4% of low-income countries were sampled in the previous study, which is extremely concerning and indicates a lag in the field of innovation that requires appropriate intervention. Interestingly, three of the studies were conducted regardless of income group classification. According to the findings, low middle-income and low-income countries require more attention, particularly in determining the factors that stifle their innovation growth.

Distribution of countries by income groups.

5. Interpreting the research objectives

This article thoroughly examines the DEA used to assess innovation efficiency. Although the concept of NIS has been around since the late 1980s, the emphasis on using NIS to measure innovation is relatively new ( Balzat and Hanusch, 2004 ; Teixeira, 2014 ; Watkins et al., 2015 ). As a result, the DEA has employed a large number of researchers to examine the performance of NIS. The majority of the articles in the review can be found in a national context ( Carayannis et al., 2016 ; Guan and Chen, 2012 ; Kontolaimou et al., 2016 ; Kou et al., 2016 ; Liu et al., 2015 ; Lu et al., 2014 ; Nasierowski and Arcelus, 2003 ; Pan et al., 2010 ), and this context is an important factor for the analysis ( Jabbour, 2013 ; Mariano et al., 2015 ). While the term “Regional Innovation System” (RIS) is derived from the term “National Innovation System,” it focuses on a specific geographical area ( Iammarino, 2005 ). Empirical research has confirmed that spatial features have a significant impact on firms' innovative performance since 2000; however, the geographical dimension has characterized economic development and is assumed to be an exogenous explanatory variable ( Iammarino, 2005 ). Geographic, regional and local conditions, as well as the general macroeconomic situation of the NIS in which the regions are embedded, play a significant role in RIS performance ( Ho, 2009 ; Muscio, 2006 ) and long-term economic growth.

In addition, certain geographic regions such as the European Union – EU ( Carayannis et al., 2016 ; Dobrzanski, 2018 ; Hudec and Prochadzkova, 2013 ; Jurickova et al., 2017 ; Matei and Aldea, 2012 ; Pinto and Pereira, 2013 ; Samara et al., 2012 ), Central and Eastern Europe – CEEC ( Bielicki and Lesniak, 2016 ; Dobrzanski, 2018 ), Eastern Europe and Central Asian – EECA ( Yesilay and Halac, 2020 ) and Association of Southeast Asia Nations – ASEAN ( Afzal et al., 2019 ) were used to measure the efficiency of NIS. Meanwhile, Guan and Chen (2012) , Kotsemir (2013) , and Kou et al. (2016) have measured the OECD countries' innovation efficiency. On the other hand, Klevenhusen et al. (2020) considered the free trade bloc to which each country belongs as a context variable, such as the North American Free Trade Agreement (NAFTA), Asia-Pacific Economic Cooperation (APEC), EU, and ASEAN. This context variable arouses the interest of other countries participating in the free trade bloc to simulate the performance of innovations. In addition, there are several studies based on a group of tie-up countries such as BRICS (Brazil, Russia, India, People's Republic of China (China), and South Africa), OECD, CEEC, EECA and ASEAN, as well as a large number of studies focusing on the different regions in the EU. Future studies involving geographic regions, free trade bloc participants, and tie-up countries are required to provide input in formulating and developing innovation policies. Table 4 shows the examined countries by geographic region, and group of tie-up countries.

The examined countries by geographic region, and tie-up countries.

These studies are expected to be conducted in developed, developing, and least developed countries, which is consistent with previous research by Lacka and Brzezicki (2021) , Sharma and Thomas (2008) , and Yesilay and Halac (2020) . Most of these studies focus on developed countries ( Choi and Zo, 2019 ; Hudec and Prochadzkova, 2013 ; Kou et al., 2016 ; Matei and Aldea, 2012 ; Rousseau and Rousseau, 1997 ; Tarnawska and Mavroeidis, 2015 ) when looking at the individual numbers because innovation is an important driver of economic growth in developed countries ( Parkey, 2012 ; Kurniawati, 2020 ). Despite the fact that the percentage of examined developing nations is higher than the percentage of developed nations, but the frequency of countries sampled in developing nations is very limited, focusing only on Bulgaria, China, Romania, Russia, and Turkey. The studies recognize the difficulties that developing countries in particular face in embarking on innovations, as resources are limited, knowledge-based is relatively weak, economic conditions in developing countries are very heterogeneous ( Choi and Zo, 2019 ) and people are not as aware and skilled ( UNCTAD, 2021 ). The NIS concept has been used in developed and, more recently, developing countries, but it is more limited in LDCs ( Metcalfe and Ramlogan, 2008 ). Overall, the literature on innovation performance has not paid attention to less developed countries. Researchers focused on allowing these countries to mark themselves in the Global Innovation Index (GII), so studies focusing on developing and least developed countries can help them improve their innovation policies.

In terms of methodological analysis, it is clear that all articles used quantitative methods to assess the efficiency of innovation. This finding demonstrated that the quantitative approach is the most commonly used methodological approach by most researchers ( Bakhtiar et al., 2021 ; Cai, 2011 ; Carayannis et al., 2016 ; Firsova and Chernyshova, 2020 ; Kryzhko et al., 2020 ; Valdez and Balderrama, 2015 ; Zhang, 2013 ). These mainly involve the calculation of country key performance indicators by using previous data available for the NIS details. The studies often look at a specific year, period of time and collect the available data against the required factors and then analyze them using the various mathematical methods with statistical tools. Researchers prefer to collect data stored in various regional or global indices related to countries' performance in terms of stimulating innovation. This data can be used by researchers to implement DEA techniques and determine required country performance metrics once the study period has been determined. For example, Jankowska et al. (2017) , Jurickova et al. (2017) , Namazi and Mohammadi (2018) used secondary data from the GII provided by the World Intellectual Property Organization (WIPO) to measure innovation efficiency at the national level. In addition, Kudryavtseva et al. (2016) and Mahroum and Al-Saleh (2013) used the scoreboard's innovation information to assess a comparative assessment of innovation levels between the European Union and Russia.

The studies examined mainly focus on the use of non-parametric methods, especially DEA, as models to measure the innovation efficiency of the considered countries. Several researchers have emphasized that DEA is one of the most effective methods to measure the efficiency of countries on NIS ( Alnafrah, 2021 ; Bielicki and Lesniak, 2016 ; Botha et al., 2016 ; Carayannis et al., 2016 ; Fotia and Teclean, 2019 ; Krstic and Mimovic, 2018 ; Namazi and Mohammadi, 2018 ; Nasierowski and Arcelus, 2003 ; San, 2011 ; Yesilay and Halac, 2020 ). The traditional model of this DEA was developed to measure the efficiency of the DMU as a whole ( Kao, 2014 ) without considering the performance of other sub-processes within the unit ( Alnafrah, 2021 ). However, while conducting the study, some evolved DEA methods were encountered, including N-DEA, SBM-DEA, DN-DEA, RN-DEA, B-DEA, and SE-DEA. The evolution of the DEA method over time allows for the inclusion of intermediate results, making it easier to measure the contribution of these units to the units' overall performance ( Shewell and Migiro, 2016 ). Table 5 lists the benefits of evolved DEA models, and future studies should incorporate such DEA models to produce more reliable results.

The advantages of various DEA models.

The overall performance levels of innovation are evaluated, and the most important components used here are the effectiveness generated in the respected country with the implementation of NIS, and the total efficiency is measured using the DEA modeling. The input and output variables in the NIS can be quantified to measure a country's overall performance, and it includes the two major components of efficiency and productivity. The discovery reveals that DEA is widely used to measure efficiency ( Alnafrah, 2021 ; Guede-Cid et al., 2021 ; Kotsemir, 2013 ) rather than productivity. Although the rate of productivity measurement is still far below efficiency, measuring both productivity and the efficiency of innovation plays an important role in providing insight and input to policymakers.

Eventually, the study reveals that only a few previous studies have taken income grouping into account. Table 6 classifies the sampled countries based on their income levels. There have been very few studies that combine different income groups. Furthermore, few such middle- and low-income countries have been studied previously, so more samples from this income group are required to understand their innovation performance. At the same time, understanding the efficiency of innovation from a global perspective necessitates a non-income group study. Choi and Zo (2019) classified each cluster's member countries by income, allowing them to find closer targets related to economic resources in benchmarks. Meanwhile, Cai (2011) reinforces that the BRICS should improve NIS efficiency and boost innovation capacity in order to sustain rapid growth and escape the middle-income trap. Furthermore, Leontitsis et al. (2018) examined the results of multidimensional efficiency analysis scores by income group. Such cross-countries comparative studies are critical for providing insight into making innovation an explicit part of future strategic plans, as well as for solidifying the importance of and accountability for innovation.

The sampled countries based on their income levels.

Another significant finding revealed that previous studies sampled approximately 56% of 217 countries, as shown in Fig. 9 . Obviously, 44% of countries have yet to be sampled in studies on innovation efficiency. Previous studies also only looked at 44% of low-income countries, 55% of high-income countries, and 60% of middle-income countries. According to these studies, the frequency of high-income countries examined is higher than the middle- and low-income groups. However, 45% of high-income countries have yet to be sampled in previous studies. So, future studies should focus on comparative analysis regardless of income group taxonomy, including countries that have never been studied previously. Subsequently, the findings can help countries, that are lagging in the field of innovation to formulate and develop appropriate innovation policies.

The distribution of nations evaluated in comparison to the total number of countries.

6. Discussion and research agenda

Based on the findings of the literature reviews, the final framework for future research is depicted in Fig. 10 . The research showed certain gaps existing in the present literature, including the measurement of innovation efficiencies using DEA. Although there have been many studies on the measurement of innovation efficiency at the national level, there is still a need to continue and increase the number of studies in cross countries. It is also important to note that the context, structure, and institutions that support innovation change over time ( Zabala-Iturriagagoitia et al., 2020 ) and volatile. So, further research to bridge the gap between global and national contexts is possible and necessary. If both the regional and sectoral innovation systems are strengthened, the innovation efficiency of the NIS can be increased. Innovation efficiency measurement based on certain geographic region, participation in free trade bloc and group of tie-up countries also would provide useful data and information to improve nation's efficiency. This is necessary in order to assess the impact of membership on each bloc's efficiency level. Comparative analysis allows countries or regions to benchmark against the ‘best in class’ innovator and improve their performance ( Shewell and Migiro, 2016 ).

Framework for future research.

The reviewed articles undoubtedly included developed countries, primarily Europe countries (in terms of number of countries). This is because Europe has easy access to comparable and reliable data on NIS development ( Kotsemir, 2013 ). The studies on developed nations highlight the measures taken by them and how they have helped them to achieve a certain position in terms of innovation (refer Fig. 11 ). Moreover, it is reassuring to see that researchers in efficiencies measures on developing economies. Many developing countries are among the most prominent prime mover in innovation, and optimal innovation implementation can ensure further growth and development. Meanwhile, as most LDCs face significant structural obstacles to long-term development, future research should concentrate on developing nations and LDCs in order to help those countries understand inherent constraints and advantages and develop effective innovation strategies to improve their future innovation performance.

Innovation performance at different income levels, 2021 ( WIPO, 2021 ).

Given that the methodology applied is largely quantitative, an attempt using the other methods can be undertaken to determine the efficiency of NIS to locate other minute factors missed under the quantitative mode of study. The availability of secondary data, such as GII, Global Competitiveness Index, World Bank and European Innovation Scorecard reports, should optimally use to measure the efficiency of innovations. These reports provide a wealth of innovation-related data and information useful for future studies. Quantitative method research on innovation performance can help a country or region analyze its position and take appropriate actions/measures to enhance its innovation policies.

Another important observation made is that the studies focusing on efficiencies of low-income countries are scarce, although several low-income countries, such as Malawi, Rwanda, and Madagascar, have been innovating since 2012 ( WIPO, 2021 ). In addition, low-income sub-Saharan African economies are also effectively converting their limited innovation inputs and resources into innovative outputs ( WIPO, 2021 ). The frequency of middle-income countries examined in previous studies, on the other hand, is limited to a few countries, including Bulgaria, China, Romania, Russia, and Turkey. However, several lower-middle income countries, including India, Vietnam, and the Philippines, are experiencing a changing innovation landscape that has the potential to change the global innovation landscape ( WIPO, 2021 ). Fig. 11 depicts the level of innovation at various income levels. Therefore, upper middle-income economies will almost certainly continue to improve their performance and innovation systems in order to compete with higher-income countries. As a result, regardless of income group classification, a comparative analysis of the existing conditions on national and regional innovation can contribute to the current body of knowledge. These studies should provide a profile of member countries' national innovation policies as well as a comparative analysis of the most significant strengths and areas for improvement.

In a future study, comparative analysis among cross-countries, particular regions, country taxonomy and income groups on national efficiency through DEA will provide insights for countries to develop and implement effective innovation strategies for maximizing innovation outcomes. Comparative analysis helps peers benchmark with the best innovators to improve the efficiency of innovation and maintain international competitiveness and sustainability. Measuring innovation capacity and output provides clarity to decision-makers in government, business, and elsewhere, as they eagerly look forward to creating policies that empower their citizens to invent and create more efficiently. Ultimately, exploring future innovation performance will help countries develop and implement their innovation strategies towards integration to stimulate economic growth. A thorough examination of the efficiency of the national innovation system from a global perspective will be beneficial in developing an innovation research agenda and strategies for innovation sustainability.

7. Conclusion

The systematic review undertaken has been initiated to provide insights into the exponentially growing innovative measures across the world. The requirement for any country to innovate becomes utmost necessary at the present time. Hence, to facilitate innovative practices, the NIS was initiated to provide support to countries that want to innovate. While researchers in the past have provided several important insights into the matter, this study attempted to bring together similar contributions to present an overview for future researchers. Therefore, this study was conducted by using a classification and coding system as the main approach. This study fills the gap as previous researchers did not pay attention to the comparative analysis and cross-countries efficiency evaluation of NIS in their review articles. A comparative analysis of cross-countries innovation efficiency would give NIS insight on how to strengthen their ability and capacity to improve innovation performance. Subsequently, the framework also put forward an agenda for future research.

7.1. Implication for theory and practice

Innovation is inevitable. Innovation can have a significant impact on a country's performance and survival. In theory, the resulting framework conforms to the evolved DEA models, and comparative analysis regardless of income group taxonomy can provide a clear picture of the global efficiency of innovation. Countries must keep up with the ever-changing environment in this modern age of technological advancement and competitiveness. To adapt to the rapid environmental changes, it is critical to close the gaps identified in this study in order to assist countries in strengthening their innovation policies and strategies. Without appropriate enhancement, innovation capacities and capabilities will deteriorate over time. So, innovation resources should be used optimally to produce the highest level of innovation output. Furthermore, this study includes a systematic literature review process, with the results potentially presenting a more comprehensive framework for future research. The inclusion of a sectoral innovation system in a different context as a comparison for future research agendas can encourage a broader study.

7.2. Limitations

This study has some limitations. The types of input and output variables used in the studies were not considered in this review. To assess the efficiency of innovation, various inputs and outputs are used. Summarizing the factors identified during the study, the most important factors influencing NIS efficiency include the input factors corresponding to financial grants, the associated staff, and the number of units involved. The revenue from these innovative practices is the most important output factor, followed by the number of patents filed. These indicators affect the efficiency of the NIS, so it is best to examine the inputs and outputs when measuring innovation efficiency.

Declarations

Author contribution statement.

All authors listed have significantly contributed to the development and the writing of this article.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement

Declaration of interests statement.

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.

• Abbasi F., Hajihoseini H., Haukka S. Use of virtual index for measuring efficiency of innovation systems: a cross-country study. Int. J. Technol. Manag. Sustain. Dev. 2011; 9 (3):195–212. [ Google Scholar ]
• Afzal M.N.I. An empirical investigation of the National Innovation System (NIS) using Data Envelopment Analysis (DEA) and the TOBIT model. Int. Rev. Appl. Econ. 2014; 28 (4):507–523. [ Google Scholar ]
• Afzal M.N.I., Lawrey R., Gope J. Understanding national innovation system (NIS) using Porter's diamond model (PDM) of competitiveness in ASEAN-05. Compet. Rev. 2019; 29 (4):336–355. [ Google Scholar ]
• Ajibefun I.A. An evaluation of parametric and non-parametric methods of technical efficiency measurement: application to small scale food crop production in Nigeria. J. Agric. Soc. Sci. 2008; 4 (3):95–100. [ Google Scholar ]
• Alnafrah I. Efficiency evaluation of BRICS's national innovation systems based on bias-corrected network data envelopment analysis. J. Innov. Entrepreneurship. 2021; 10 (1):1–28. [ Google Scholar ]
• Amui L.B.L., Jabbour C.J.C., de Sousa Jabbour A.B.L., Kannan D. Sustainability as a dynamic organizational capability: a systematic review and a future agenda toward a sustainable transition. J. Clean. Prod. 2017; 142 :308–322. [ Google Scholar ]
• Andersen P., Petersen N.C. A procedure for ranking efficient units in data envelopment analysis. Manag. Sci. 1993; 39 (10):1261–1264. [ Google Scholar ]
• Anderson H.J., Stejskal J. Diffusion efficiency of innovation among EU member states: a data envelopment analysis. Economies. 2019; 7 (2):34. [ Google Scholar ]
• Aparicio J., Ortiz L., Pastor J.T., Mikel Zabala-Iturriagagoitia J. Introducing crossproductivity: a new approach for ranking productive units over time in data envelopment analysis. Comput. Ind. Eng. 2020; 144 [ Google Scholar ]
• Archibugi D. National innovation systems. A comparative analysis: Richard R. Nelson (Editor), (Oxford University Press, New York and London, 1993) xii + 541 pp. Res. Policy. 1996; 25 (5):838–842. [ Google Scholar ]
• Asmare E., Begashaw A. Review on parametric and nonparametric methods of efficiency analysis. Open Access Bioinform. 2018; 2 (2):1–7. [ Google Scholar ]
• Aviles-Sacoto S.V., Cook W.D., Güemes-Castorena D., Zhu J. Modelling efficiency in regional innovation systems: a two-stage data envelopment analysis problem with shared outputs within groups of decision-making units. Eur. J. Oper. Res. 2020; 287 (2):572–582. [ Google Scholar ]
• Bakhtiar A., Ghazinoory S.S., Aslani A., Mafi V. Efficiency-effectiveness assessment of national innovation systems: comparative analysis. J. Sci. Technol. Policy Manag. 2021 [ Google Scholar ]
• Balzat M., Hanusch H. Recent trends in the research on national innovation systems. J. Evol. Econ. 2004; 14 (2):197–210. [ Google Scholar ]
• Banker R., Emrouznejad A., Bal H., Alp I., Cengiz M. Data envelopment analysis and performance measurement. Proceedings of the 11th International Conference of DEA; Samsun, Turkey, June 2013; 2013. [ Google Scholar ]
• Belitz H., Clemens M., von Hirschhausen C., SchmidtEhmcke J., Werwatz A., Zloczysti P. An indicator for national systems of innovation – methodology and application to industrialized countries. 2011. Discussion Paper.
• Bielicki M., Lesniak M. Proceedings of the 11th European Conference on Innovation and Entrepreneurship. 2016. CEE cross-country comparison of national innovation systems efficiency: DEA approach. [ Google Scholar ]
• Botha L., Grobbelaar S., Bam W. Towards a framework to guide the evaluation of innovation systems. S. Afr. J. Ind. Eng. 2016; 27 (3):64–78. [ Google Scholar ]
• Bouwman H., Nikou S., de Reuver M. Digitalization, business models, and SMEs: how do business model innovation practices improve performance of digitalizing SMEs? Telecommun. Policy. 2019; 43 (9) [ Google Scholar ]
• Bresciani S., Puertas R., Ferraris A., Santoro G. Innovation, environmental sustainability and economic development: DEA-Bootstrap and multilevel analysis to compare two regions. Technol. Forecast. Soc. Change. 2021; 172 (C) [ Google Scholar ]
• Broekel T., Rogge N., Brenner T. The innovation efficiency of German regions – a shared-input DEA approach. Rev. Reg. Res. 2017; 38 (1):77–109. [ Google Scholar ]
• Cai Y. Factors affecting the efficiency of the BRICSs' national innovation systems: a comparative study based on DEA and panel data analysis. 2011. Economics Discussion Papers, Kiel Institute for the World Economy (IfW)
• Carayannis E.G., Goletsis Y., Grigoroudis E. Multi-level multi-stage efficiency measurement: the case of innovation systems. Oper. Res. 2015; 15 (2):253–274. [ Google Scholar ]
• Carayannis E.G., Grigoroudis E., Goletsis Y. A multilevel and multistage efficiency evaluation of innovation systems: a multiobjective DEA approach. Expert Syst. Appl. 2016; 62 (C):63–80. [ Google Scholar ]
• Centobelli P., Cerchione R., Esposito E. Environmental sustainability in the service industry of transportation and logistics service providers: systematic literature review and research directions. Transp. Res., Part D, Transp. Environ. 2017; 53 :454–470. [ Google Scholar ]
• Chang C-C. Influences of knowledge spillover and utilization on the NIS performance: a multi-stage efficiency perspective. Qual. Quant. 2014; 49 (5):1945–1967. [ Google Scholar ]
• Charnes A., Cooper W.W., Rhodes E. Measuring the efficiency of decision making units. Eur. J. Oper. Res. 1978; 2 (6):429–444. [ Google Scholar ]
• Chen C.P., Hu J.L., Yang C.H. An international comparison of R&D efficiency of multiple innovative outputs: the role of the national innovation system. Innovation. 2011; 13 (3):341–360. [ Google Scholar ]
• Chen K., Guan J. Measuring the efficiency of China's regional innovation systems: application of network data envelopment analysis (DEA) Reg. Stud. 2012; 46 (3):355–377. [ Google Scholar ]
• Chen P-C., Hung S-W. An actor-network perspective on evaluating the R&D linking efficiency of innovation ecosystems. Technol. Forecast. Soc. Change. 2016; 112 :303–312. [ Google Scholar ]
• Choi H., Zo H. Assessing the efficiency of national innovation systems in developing countries. Sci. Public Policy. 2019; 46 (4):530–540. [ Google Scholar ]
• Cooper W.W., Seiford L.M., Tone K. Springer; New York, NY, USA: 2006. Introduction to Data Envelopment Analysis and Its Uses with DEA-Solver and References. [ Google Scholar ]
• Cooper W.W., Seiford L.M., Tone K. second edition. Springer-Verlag; US: 2007. Data Envelopment Analysis. [ Google Scholar ]
• Dahlman C.J. Technology strategy in East Asian developing countries. J. Asian Econ. 1994; 5 (4):541–572. [ Google Scholar ]
• Deng J., Zhang N., Ahmad F., Draz M.U. Local government competition, environmental regulation intensity and regional innovation performance: an empirical investigation of Chinese Provinces. Int. J. Environ. Res. Public Health. 2019; 16 (12):2130. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Dereli D.D. Innovation management in global competition and competitive advantage. Proc., Soc. Behav. Sci. 2015; 195 :1365–1370. [ Google Scholar ]
• Didenko A., Loseva O., Abdikeev N. Measuring efficiency of regional innovation system with DEA and PCA. 2017 IEEE 11th International Conference on Application of Information and Communication Technologies; AICT; 2017. [ Google Scholar ]
• Diez J.R., Kiese M. In: International Encyclopedia of Human Geography. Kitchin R., Thrift N., editors. Elsevier; Oxford: 2009. Regional innovation systems; pp. 246–251. [ Google Scholar ]
• Dobrzanski P. Innovation expenditures efficiency in Central and Eastern European countries. Zb. Rad. Ekon. Fak. Rij. 2018; 36 (2):827–859. [ Google Scholar ]
• Dobrzanski P., Bobowski S., Chrysostome E., Velinov E., Strouhal J. Toward innovation-driven competitiveness across African countries: an analysis of efficiency of R&D expenditures. J. Compet. 2021; 13 (1):5–22. [ Google Scholar ]
• Dodgson M., Gann D., Salter A. The role of technology in the shift towards open innovation: the case of Procter & Gamble. R & D Manag. 2006; 36 (3):333–346. [ Google Scholar ]
• Edquist C. The Oxford Handbook of Innovation. 2009. Systems of innovation: perspectives and challenges. [ Google Scholar ]
• Edquist C., Zabala-Iturriagagoitia J.M. Assessing the performance of national innovation systems in Europe. 21st International Conference on Science and Technology Indicators; Spain; 2016. [ Google Scholar ]
• Edquist C., Zabala-Iturriagagoitia J.M., Barbero J., Zofío J.L. On the meaning of innovation performance: is the synthetic indicator of the Innovation Union Scoreboard flawed? Res. Eval. 2018:1–16. [ Google Scholar ]
• Edler J., Fagerberg J. Innovation policy: what, why, and how. Oxf. Rev. Econ. Policy. 2017; 33 (1):2–23. [ Google Scholar ]
• Fang J.-W., Chiu Y. Research on innovation efficiency and technology gap in China economic development. Asia-Pac. J. Oper. Res. 2017; 34 (2) (1-22) [ Google Scholar ]
• Fare R., Whittaker G. An intermediate input model of dairy production using complex survey data. J. Agric. Econ. 1995; 46 (2):201–213. [ Google Scholar ]
• Fare R., Grosskopf S. Network DEA. Socio-Econ. Plan. Sci. 2000; 34 (1):35–49. [ Google Scholar ]
• Farrell M.J. The measurement of productive efficiency. J. R. Stat. Soc. 1957; 120 (3):253–290. [ Google Scholar ]
• Firsova A., Chernyshova G. Efficiency analysis of regional innovation development based on DEA Malmquist Index. Information. 2020; 11 (6):294. [ Google Scholar ]
• Freeman, C., 1982. Technological infrastructure and international competitiveness. Draft paper submitted to the OECD Ad hoc group on Science, Technology and Competitiveness, August 1982. Mimeo. Freeman C. Technological infrastructure and international competitiveness. Ind. Corp. Change. 2004; 13 :541–569. Later published as. [ Google Scholar ]
• Freeman C. Pinter Publishers; London/New York: 1987. Technology and Economic Performance: Lessons from Japan. [ Google Scholar ]
• Foddi M., Usai S. Regional knowledge performance in Europe. Growth Change. 2013; 44 (2):258–286. [ Google Scholar ]
• Fotia A., Teclean C. The innovation efficiency in Central and Eastern Europe – an input-output comparative analysis between Czech Republic, Hungary, Poland and Romania. EURINT, Cent. Eur. Stud., Alexandru Ioan Cuza Univ. 2019; 6 (1):269–287. [ Google Scholar ]
• Godinho F.M., Veloso S.E. From time-based competition (TBC) to quick response manufacturing (QRM): the evolution of research aimed at lead time reduction. Int. J. Adv. Manuf. Technol. 2013; 64 :5–8. [ Google Scholar ]
• Gokhberg L., Roud V. Structural changes in the national innovation system: longitudinal study of innovation modes in the Russian industry. Econ. Change Restruct. 2016; 49 :269–288. [ Google Scholar ]
• Golichenko O.G. The national innovation system. Probl. Econ. Transit. 2016; 58 (5):463–481. [ Google Scholar ]
• Grilo A., Santos J. Measuring efficiency and productivity growth of new technology-based firms in business incubators: the Portuguese case study of Madan Parque. Sci. World J. 2015; 2015 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Guan J., Chen K. Modeling macro-R&D production frontier performance: an application to Chinese province-level R&D. Scientometrics. 2010; 82 (1):165–173. [ Google Scholar ]
• Guan J., Chen K. Modeling the relative efficiency of national innovation systems. Res. Policy. 2012; 41 (1):102–115. [ Google Scholar ]
• Guan J., Zuo K. A cross-country comparison of innovation efficiency. Scientometrics. 2014; 100 (2):541–575. [ Google Scholar ]
• Guede-Cid R., Rodas-Alfaya L., Leguey-Galán S., Cid-Cid A.I. Innovation efficiency in the Spanish service sectors, and open innovation. J. Open Innov. Technol. Market Complex. 2021; 7 (1):62. [ Google Scholar ]
• Gunay E.N., Kazazoglu G.R. Palgrave Macmillan Books, Palgrave Macmillan; 2016. National Innovation Efficiency During the Global Crisis. [ Google Scholar ]
• Halaskova M., Gavurova B., Kocisova K. Research and development efficiency in public and private sectors: an empirical analysis of EU countries by using DEA methodology. Sustainability. 2020; 12 (17):7050. [ Google Scholar ]
• Ho M.H-C. How regional innovation systems play a relative competitive role within knowledge networks. Eur. Plan. Stud. 2009; 17 (12):1881–1905. [ Google Scholar ]
• Hudec O., Prochadzkova M. The relative efficiency of knowledge innovation processes in EU countries. Stud. Reg. Sci. 2013; 43 (1):145–162. [ Google Scholar ]
• Hsu Y. Cross national comparison of innovation efficiency and policy application. Afr. J. Bus. Manag. 2011; 5 (4):1378–1387. [ Google Scholar ]
• Iammarino S. An evolutionary integrated view of regional systems of innovation: concepts, measures and historical perspectives. Eur. Plan. Stud. 2005; 13 (4):497–519. [ Google Scholar ]
• Ivanova E., Cepel M. The impact of innovation performance on the competitiveness of the Visegrad 4 countries. J. Compet. 2018; 10 (1):54–72. [ Google Scholar ]
• Jabbour C.J.C. Environmental training in organisations: from a literature review to a framework for future research. Resour. Conserv. Recycl. 2013; 74 :144–155. [ Google Scholar ]
• Jaloudi M.M. The efficiency of Jordan insurance companies and its determinants using DEA, slacks, and logit models. J. Asian Bus. Econ. Stud. 2019; 26 (1):153–166. [ Google Scholar ]
• Jankowska B., Matysek-Jędrych A., Mroczek-Dąbrowska K. Efficiency of national innovation systems: Poland and Bulgaria in the context of the global innovation index. Comp. Econ. Res. 2017; 20 (3):77–94. [ Google Scholar ]
• Jurickova E., Bobak R., Pilik M., Hrusecka D. Efficiency of innovation system in the Czech Republic: comparison with other European countries. New Trends Issues Proc. Humanit. Soc. Sci. 2017; 4 (10):90–96. [ Google Scholar ]
• Kaihua C., Mingting K. Staged efficiency and its determinants of regional innovation systems: a two-step analytical procedure. Ann. Reg. Sci. 2014; 52 (2):627–657. [ Google Scholar ]
• Kao C., Hwang S.N. Efficiency decomposition in two-stage data envelopment analysis: an application to non-life insurance companies in Taiwan. Eur. J. Oper. Res. 2008; 185 (1):418–429. [ Google Scholar ]
• Kao C. Network data envelopment analysis: a review. Eur. J. Oper. Res. 2014; 239 (1):1–16. [ Google Scholar ]
• Klevenhusen A., Coelho J., Warszawski L. Innovation efficiency in OECD countries: a non-parametric approach. J. Knowl. Econ. 2020; 12 (3):1064–1078. [ Google Scholar ]
• Koellinger P. The relationship between technology, innovation, and firm performance—empirical evidence from e-business in Europe. Res. Policy. 2008; 37 (8):1317–1328. [ Google Scholar ]
• Kong M., Wang X., Wu Q. The development efficiency of China's innovative industrial clusters-based on the DEA-Malmquist model. Arab. J. Geosci. 2021; 14 (7):1–15. [ Google Scholar ]
• Kontolaimou A., Giotopoulos I., Tsakanikas A. A typology of European countries based on innovation efficiency and technology gaps: the role of early-stage entrepreneurship. Econ. Model. 2016; 52 (PB):477–484. [ Google Scholar ]
• Kotsemir M.N. Higher School of Economics Research Papers, Paper no. WP BRP 16/STI/2013. National Research University Higher School of Economics; 2013. Measuring national innovation systems efficiency: a review of DEA approach. [ Google Scholar ]
• Kou M., Chen K., Wang S., Shao Y. Measuring efficiencies of multi-period and multi-division systems associated with DEA: an application to OECD countries' national innovation systems. Expert Syst. Appl. 2016; 46 (C):494–510. [ Google Scholar ]
• Kurniawati M.A. The role of ICT infrastructure, innovation and globalization on economic growth in OECD countries, 1996–2017. J. Sci. Technol. Policy Manag. 2020; 11 (2):193–215. [ Google Scholar ]
• Krstic A., Mimovic P. Measuring efficiency of the Serbian national innovation model: DEA approach. Contemporary Issues in Economics, Business and Management; EBM; 2018. [ Google Scholar ]
• Kryzhko D., Rudskaya I., Skhvediani A., Alamshoev A. Evaluation of technical efficiency of regional innovation system on the basis of DEA modeling. Proceedings of the 2nd International Scientific Conference on Innovations in Digital Economy; SPBPU IDE-2020; New York, NY, USA: Association for Computing Machinery; 2020. [ Google Scholar ]
• Kudryavtseva S.S., Shinkevich A.I., Ostanina S.S., Vodolazhskaya E.L., Chikisheva N.M., Lushchik I.V., Shirokova L.V., Khairullina E.R. The methods of national innovation systems assessing. Int. Rev. Manag. Mark. 2016; 6 (2S):225–230. [ Google Scholar ]
• Kuhlmann S., Ordóñez-Matamoros G., editors. Research Handbook on Innovation Governance for Emerging Economies. Towards Better Models. Edward Elgar Publishing; Cheltenham, UK: 2017. [ Google Scholar ]
• Lacka I., Brzezicki L. The efficiency and productivity evaluation of national innovation systems in Europe. Eur. Res. Stud. J. 2021; XXIV (3):471–496. [ Google Scholar ]
• Lafuente E., Szerb L., Zoltan J. Country level efficiency and national systems of entrepreneurship: a data envelopment analysis approach. J. Technol. Transf. 2016; 41 (6):1260–1283. [ Google Scholar ]
• Lala K., Sinha K. Role of technology incubation in India's innovation system: a case of the Indian Institute of Technology Kanpur Incubation Centre. Millenn. Asia. 2019; 10 (1):91–110. [ Google Scholar ]
• Leontitsis A., Philippas D., Sickles R., Tziogkidis P. Rice University, Department of Economics; 2018. Evaluating countries' innovation potential: an international perspective. Working Papers 18-011. [ Google Scholar ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Sci. J. 2006; 9 :182–212. [ Google Scholar ]
• Li R., Li Y., Cui Z. Application of data envelopment analysis to efficiency evaluation on R&D input and output. Open Autom. Control Syst. J. 2014; 6 :194–199. [ Google Scholar ]
• Liberati A., Altman D.G., Tetzlaff J., Mulrow C., Gøtzsche P.C., Ioannidis J.P.A., Clarke M., Devereaux P.J., Kleijnen J., Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009; 6 (7) [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Lin T.-Y., Chiu S.-H., Yang H.-L. Performance evaluation for regional innovation systems development in China based on the two-stage SBM-DNDEA model. Socio-Econ. Plan. Sci. 2021 [ Google Scholar ]
• Liu M.C., Wang J.C., Wu M.T. Typology and knowledge productivity of regional innovation system: evidence from China. J. Bus. Econ. 2014; 5 (1):49–60. [ Google Scholar ]
• Liu J.S., Lu W.-M., Ho M.H.-C. National characteristics: innovation systems from the process efficiency perspective. R & D Manag. 2015; 45 (4):317–338. [ Google Scholar ]
• Liu Z., Chen X., Chu J., Zhu Q. Industrial development environment and innovation efficiency of high-tech industry: analysis based on the framework of innovation systems. Technol. Anal. Strateg. Manag. 2017; 30 (4):434–446. [ Google Scholar ]
• Lu W.M., Kweh Q.L., Huang C.L. Intellectual capital and national innovation systems performance. Knowl.-Based Syst. 2014; 71 (1):201–210. [ Google Scholar ]
• Lundvall B.-Å. Aalborg University Press; Aalborg: 1985. Product Innovation and User-Producer Interaction. [ Google Scholar ]
• Lundvall B.-A. National innovation systems-analytical concept and development tool. Ind. Innov. 2007; 14 (1):95–119. [ Google Scholar ]
• Lundvall B-A. vol. 2. Anthem Press; 2010. National Systems of Innovation: Toward a Theory of Innovation and Interactive Learning. (Anthem Other Canon Economics Series). [ Google Scholar ]
• Lundvall B.-A. The Learning Economy and the Economics of Hope. Anthem Press; 2016. National innovations systems and globalisation; pp. 351–374. Chapter 13. [ Google Scholar ]
• Mahroum S., Al-Saleh Y. Towards a functional framework for measuring national innovation efficacy. Technovation. 2013; 33 (10–11):320–332. [ Google Scholar ]
• Mariano E.B., Sobreiro V.A., do Nascimento Rebelatto D.A. Human development and data envelopment analysis: a structured literature review. Omega. 2015; 54 (C):33–49. [ Google Scholar ]
• Matei M.M., Aldea A. Ranking national innovation systems according to their technical efficiency. Proc., Soc. Behav. Sci. 2012; 62 :968–974. [ Google Scholar ]
• Metcalfe S., Ramlogan R. Innovation systems and the competitive process in developing economies. Q. Rev. Econ. Finance. 2008; 48 (2):433–446. [ Google Scholar ]
• Min S., Kim J., Sawng Y.W. The effect of innovation network size and public R&D investment on regional innovation efficiency. Technol. Forecast. Soc. Change. 2020; 155 [ Google Scholar ]
• Murillo-Zamorano L.R., Vega-Cervera J.A. The use of parametric and non-parametric frontier methods to measure the productive efficiency in the industrial sector: a comparative study. Int. J. Prod. Econ. 2001; 69 (3):265–275. [ Google Scholar ]
• Muscio A. From regional innovation systems to local innovation systems: evidence from Italian industrial districts. Eur. Plan. Stud. 2006; 6 :773–789. [ Google Scholar ]
• Namazi M., Mohammadi E. Natural resource dependence and economic growth: a TOPSIS/DEA analysis of innovation efficiency. Resour. Policy. 2018; 59 (C):544–552. [ Google Scholar ]
• Nasierowski W. Technical efficiency of innovations in Poland from EU perspective. Pol. J. Econ. 2019; 300 (4):79–104. [ Google Scholar ]
• Nasierowski W., Arcelus F.J. On the efficiency of national innovation systems. Socio-Econ. Plan. Sci. 2003; 37 (3):215–234. [ Google Scholar ]
• Nasierowski W., Arcelus F.J. About efficiency of innovations: what can be learned from the Innovation Union Scoreboard Index. Proc., Soc. Behav. Sci. 2012; 58 :792–801. [ Google Scholar ]
• Nelson R.R., editor. National Innovation Systems. A Comparative Analysis. Oxford University Press; New York/Oxford: 1993. [ Google Scholar ]
• OECD . OECD; Paris: 2008. OECD Reviews of Regional Innovation: North of England, UK. [ Google Scholar ]
• Ozkan E.N.G., Kazazoglu G.N. Performance of national innovation systems during the global crisis: a cross-country analysis. PressAcademia Procedia. 2016; 2 (1):37–48. [ Google Scholar ]
• Pan T.-W., Hung S.-W., Lu W.-M. DEA performance measurement of the national innovation system in Asia and Europe. Asia-Pac. J. Oper. Res. 2010; 27 (03):369–392. [ Google Scholar ]
• Park K. Evaluating innovation capacities of Chinese provincial governments. The public sphere. J. Public Policy. 2021; 9 (1):53–81. [ Google Scholar ]
• Parkey M. Clemson University. TigerPrints; 2012. Assessing the national innovation system in a developing country context: a framework and evidence from Thailand. [ Google Scholar ]
• Pinto H., Pereira S.T. Efficiency of innovation systems in Europe: an institutional approach to the diversity of national profiles. Eur. Plan. Stud. 2013; 21 (6):755–779. [ Google Scholar ]
• Popay J., Roberts H., Sowden A., Petticrew M., Arai L., Rodgers M. 2006. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews. A Product from the ESRC Methods Programme. Version 1. ESRC. [ Google Scholar ]
• Prokop V., Hajek P., Stejskal J. Configuration paths to efficient national innovation ecosystems. Technol. Forecast. Soc. Change. 2021; 168 (C) [ Google Scholar ]
• Rajapathirana R.P.J., Hui Y. Relationship between innovation capability, innovation type, and firm performance. J. Innov. Knowl. 2018; 3 (1):44–55. [ Google Scholar ]
• Ribeiro V.M., Varum C., Daniel A.D. Introducing microeconomic foundation in data envelopment analysis: effects of the ex ante regulation principle on regional performance. J. Knowl. Econ. 2020; 12 :1215–1244. [ Google Scholar ]
• Roman M. 2010. Regional efficiency of knowledge economy in the new EU countries: the Romanian and Bulgarian case. Munich Persona RePEc Archive, Romania. [ Google Scholar ]
• Rousseau S., Rousseau R. Data envelopment analysis as a tool for constructing scientometric indicators. Scientometrics. 1997; 40 (1):45–56. [ Google Scholar ]
• Salim N., Ab Rahman M.N., Abd Wahab D. A systematic literature review of internal capabilities for enhancing eco-innovation performance of manufacturing firms. J. Clean. Prod. 2019; 209 :1445–1460. [ Google Scholar ]
• Samara E., Georgiadis P., Bakouros I. The impact of innovation policies on the performance of national innovation systems: a system dynamics analysis. Technovation. 2012; 32 (11):624–638. [ Google Scholar ]
• San R.J.C. A multi criteria data envelopment analysis model to evaluate the efficiency of the renewable energy technologies. Renew. Energy. 2011; 36 (10):2742–2746. [ Google Scholar ]
• Schrempf B., Kaplan D., Schroeder D. National, regional, and sectoral systems of innovation – an overview. 2013. progressproject.eu Report for FP7 Project “Progress”.
• Sharma S., Thomas V.J. Inter-country R&D efficiency analysis: an application of data envelopment analysis. Scientometrics. 2008; 76 (3):483–501. [ Google Scholar ]
• Shewell P., Migiro S. Data envelopment analysis in performance measurement: a critical analysis of the literature. Probl. Perspect. Manag. 2016; 14 (3–3):705–713. [ Google Scholar ]
• Shin J., Kim C., Yang H. The effect of sustainability as innovation objectives on innovation efficiency. Sustainability. 2018; 10 (6) [ Google Scholar ]
• Silva W., Kimura H., Sobreiro V.A. An analysis of the literature on systemic financial risk: a survey. J. Financ. Stab. 2017; 28 (C):91–114. [ Google Scholar ]
• Simar L.O.L., Wilson P.W. Sensitivity analysis of efficiency scores: how to bootstrap in nonparametric frontier models? Manag. Sci. 1998; 44 (1):49–61. [ Google Scholar ]
• Simar L., Wilson P.W. A general methodology for bootstrapping in non-parametric frontier models. J. Appl. Stat. 2000; 27 (6):779–802. [ Google Scholar ]
• Snyder H. Literature review as a research methodology: an overview and guidelines. J. Bus. Res. 2019; 104 (C):333–339. [ Google Scholar ]
• Tasnim N., Afzal M.N.I. An empirical investigation of country level efficiency and national systems of entrepreneurship using Data Envelopment Analysis (DEA) and the TOBIT model. J. Glob. Entrep. Res. 2018 [ Google Scholar ]
• Tarnawska K., Mavroeidis V. Efficiency of the knowledge triangle policy in the EU member states: DEA approach. Triple Helix. 2015; 2 (1):1–22. [ Google Scholar ]
• Teixeira A. Evolution, roots and influence of the literature on national systems of innovation: a bibliometric account. Camb. J. Econ. 2014; 38 (1):181–214. [ Google Scholar ]
• Tidd J., Bessant J. 7th edition. John Wiley & Sons; Chichester: 2020. Managing Innovation: Integrating Technological, Market, and Organizational Change. [ Google Scholar ]
• Todttling F., Trippl M. One size fits all? Towards a differentiated regional innovation policy approach. Res. Policy. 2005; 34 (8):1203–1219. [ Google Scholar ]
• Tone K. A slacks-based measure of efficiency in data envelopment analysis. Eur. J. Oper. Res. 2001; 130 (3):498–509. [ Google Scholar ]
• Tone K. In: Handbook on Data Envelopment Analysis. Cooper W.W., Seiford L.M., Zhu J., editors. Springer; New York: 2011. Slacks-based measure of efficiency. [ Google Scholar ]
• Tone K., Tsutsui M. Dynamic DEA with network structure: a slacks-based measure approach. Omega. 2014; 42 (1):124–131. [ Google Scholar ]
• Tranfield D., Denyer D., Smart P. Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br. J. Manag. 2003; 14 (3):207–222. [ Google Scholar ]
• United Nations World economic situation and prospects. 2019. https://www.un.org/development/desa/dpad/wpcontent/uploads/sites/45/WESP2019_BOOK-ANNEX-en.pdf United Nations.
• UNCTAD . United Nations Conference on Trade and Development. 2021. Catching technological waves innovation with equity. Technology and Innovation Report 2021. [ Google Scholar ]
• Valdez L.C., Balderrama J.I.L. Efficiency of Mexico's regional innovation systems: an evaluation applying data envelopment analysis (DEA) Afr. J. Sci. Technol. Innov. Develop. 2015; 7 (1):36–44. [ Google Scholar ]
• Varblane U., Dyker D., Tamm D., von Tunzelmann N. Can the national innovation systems of the new EU member states be improved? Post-Communist Econ. 2007; 19 (4):399–416. [ Google Scholar ]
• Villaverde D., Paredes A., de los Angeles J. Assessment of innovation factors that impact competitiveness in the Mexican state entities through DEA analysis. Strategy Technol. Soc. 2020; 10 :11–35. [ Google Scholar ]
• Wang P., Lin X., Li S. Evaluation of government-supported provincial innovation competitiveness—evidence from China. Growth Change. 2019; 50 (2):587–608. [ Google Scholar ]
• Wang D., Zhao L., Yang F., Chen K. Performance evaluation of the Chinese high-tech industry: a two-stage DEA approach with feedback and shared resource. J. Ind. Manag. Optim. 2021 [ Google Scholar ]
• Wang Q., Tang J., Choi G. A two-stage eco-efficiency evaluation of China's industrial sectors: a dynamic network data envelopment analysis (DNDEA) approach. Process Saf. Environ. Prot. 2021; 148 :879–892. [ Google Scholar ]
• Watkins A., Papaioannou T., Mugwagwa J., Kale D. National innovation systems and the intermediary role of industry associations in building institutional capacities for innovation in developing countries: a critical review of the literature. Res. Policy. 2015; 44 (8):1407–1418. [ Google Scholar ]
• WIPO . World Intellectual Property Organization; Geneva: 2021. Global Innovation Index 2021: Tracking Innovation Through the COVID-19 Crisis. [ Google Scholar ]
• World Bank New World Bank country classifications by income level: 2021–2022. 2019. https://blogs.worldbank.org/opendata/new-world-bank-country-classifications-income-level-2021-2022
• Xiong X., Yang G., Guan Z. A parallel DEA-based method for evaluating parallel independent subunits with heterogeneous outputs. J. Informetr. 2020; 14 (3) [ Google Scholar ]
• Xu L., Cheng M. A study on Chinese regional scientific innovation efficiency with a perspective of synergy degree. Technol. Invest. 2013; 4 (4):229–235. [ Google Scholar ]
• Yesilay R.B., Halac U. Contemporary Studies in Economic and Financial Analysis. Emerald; Bingley, UK: 2020. An assessment of innovation efficiency in EECA countries using the DEA method; pp. 203–215. [ Google Scholar ]
• Yun Z., Yuan Z., Shichan F., Luyi C. The evolution and influencing factors of national innovation efficiency from the perspective of open economy. J. Ind. Technol. Econ. 2019 [ Google Scholar ]
• Zabala-Iturriagagoitia J.M., Aparicio J., Ortiz L., Carayannis E.G., Grigoroudis E. The productivity of national innovation systems in Europe: catching up or falling behind? Technovation. 2020; 102 (C) [ Google Scholar ]
• Zemtsov S., Kotsemir M. An assessment of regional innovation system efficiency in Russia: the application of the DEA approach. Scientometrics. 2019; 120 (2):375–404. [ Google Scholar ]
• Zeng J., Ribeiro-Soriano D., Ren J. Innovation efficiency: a bibliometric review and future research agenda. Asia Pac. Bus. Rev. 2021; 27 (2):1–20. [ Google Scholar ]
• Zhang J. International comparison of national innovation system efficiency. 2013. www.techmonitor.net/tm/images/d/da/13apr_jun_sf2.pdf Tech Monitor. April–June: 23–29.
• Zhang Y., Cui M. Determining the innovation efficiency of resource-based cities using a relational network DEA model: evidence from China. Extr. Ind. Soc. 2020; 7 (4):1557–1566. [ Google Scholar ]
• Zhang C., Wang X. The influence of ICT-driven innovation: a comparative study on national innovation efficiency between developed and emerging countries. Behav. Inf. Technol. 2019 [ Google Scholar ]