artificial intelligence and machine learning research papers

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 16 October 2023

Forecasting the future of artificial intelligence with machine learning-based link prediction in an exponentially growing knowledge network

• Mario Krenn   ORCID: orcid.org/0000-0003-1620-9207 1 ,
• Lorenzo Buffoni 2 ,
• Bruno Coutinho 2 ,
• Sagi Eppel 3 ,
• Jacob Gates Foster 4 ,
• Andrew Gritsevskiy   ORCID: orcid.org/0000-0001-8138-8796 3 , 5 , 6 ,
• Harlin Lee   ORCID: orcid.org/0000-0001-6128-9942 4 ,
• Yichao Lu   ORCID: orcid.org/0009-0001-2005-1724 7 ,
• João P. Moutinho 2 ,
• Nima Sanjabi   ORCID: orcid.org/0009-0000-6342-5231 8 ,
• Rishi Sonthalia   ORCID: orcid.org/0000-0002-0928-392X 4 ,
• Ngoc Mai Tran 9 ,
• Francisco Valente   ORCID: orcid.org/0000-0001-6964-9391 10 ,
• Yangxinyu Xie   ORCID: orcid.org/0000-0002-1532-6746 11 ,
• Rose Yu 12 &
• Michael Kopp 6  

Nature Machine Intelligence volume  5 ,  pages 1326–1335 ( 2023 ) Cite this article

25k Accesses

3 Citations

1060 Altmetric

Metrics details

• Complex networks
• Computer science
• Research data

A tool that could suggest new personalized research directions and ideas by taking insights from the scientific literature could profoundly accelerate the progress of science. A field that might benefit from such an approach is artificial intelligence (AI) research, where the number of scientific publications has been growing exponentially over recent years, making it challenging for human researchers to keep track of the progress. Here we use AI techniques to predict the future research directions of AI itself. We introduce a graph-based benchmark based on real-world data—the Science4Cast benchmark, which aims to predict the future state of an evolving semantic network of AI. For that, we use more than 143,000 research papers and build up a knowledge network with more than 64,000 concept nodes. We then present ten diverse methods to tackle this task, ranging from pure statistical to pure learning methods. Surprisingly, the most powerful methods use a carefully curated set of network features, rather than an end-to-end AI approach. These results indicate a great potential that can be unleashed for purely ML approaches without human knowledge. Ultimately, better predictions of new future research directions will be a crucial component of more advanced research suggestion tools.

Similar content being viewed by others

Learning on knowledge graph dynamics provides an early warning of impactful research

TrendyGenes, a computational pipeline for the detection of literature trends in academia and drug discovery

Accelerating science with human-aware artificial intelligence

The corpus of scientific literature grows at an ever-increasing speed. Specifically, in the field of artificial intelligence (AI) and machine learning (ML), the number of papers every month is growing exponentially with a doubling rate of roughly 23 months (Fig. 1 ). Simultaneously, the AI community is embracing diverse ideas from many disciplines such as mathematics, statistics and physics, making it challenging to organize different ideas and uncover new scientific connections. We envision a computer program that can automatically read, comprehend and act on AI literature. It can predict and suggest meaningful research ideas that transcend individual knowledge and cross-domain boundaries. If successful, it could greatly improve the productivity of AI researchers, open up new avenues of research and help drive progress in the field.

The doubling rate of papers per month is roughly 23 months, which might lead to problems for publishing in these fields, at some point. The categories are cs.AI, cs.LG, cs.NE and stat.ML.

In this work, we address the ambitious vision of developing a data-driven approach to predict future research directions 1 . As new research ideas often emerge from connecting seemingly unrelated concepts 2 , 3 , 4 , we model the evolution of AI literature as a temporal network. We construct an evolving semantic network that encapsulates the content and development of AI research since 1994, with approximately 64,000 nodes (representing individual concepts) and 18 million edges (connecting jointly investigated concepts).

We use the semantic network as an input to ten diverse statistical and ML methods to predict the future evolution of the semantic network with high accuracy. That is, we can predict which combinations of concepts AI researchers will investigate in the future. Being able to predict what scientists will work on is a first crucial step for suggesting new topics that might have a high impact.

Several methods were contributions to the Science4Cast competition hosted by the 2021 IEEE International Conference on Big Data (IEEE BigData 2021). Broadly, we can divide the methods into two classes: methods that use hand-crafted network-theoretical features and those that automatically learn features. We found that models using carefully hand-crafted features outperform methods that attempt to learn features autonomously. This (somewhat surprising) finding indicates a great potential for improvements of models free of human priors.

Our paper introduces a real-world graph benchmark for AI, presents ten methods for solving it, and discusses how this task contributes to the larger goal of AI-driven research suggestions in AI and other disciplines. All methods are available at GitHub 5 .

Semantic networks

The goal here is to extract knowledge from the scientific literature that can subsequently be processed by computer algorithms. At first glance, a natural first step would be to use large language model (such as GPT3 6 , Gopher 7 , MegaTron 8 or PaLM 9 ) on each article to extract concepts and their relations automatically. However, these methods still struggle in reasoning capabilities 10 , 11 ; thus, it is not yet directly clear how these models can be used for identifying and suggesting new ideas and concept combinations.

Rzhetsky et al. 12 pioneered an alternative approach, creating semantic networks in biochemistry from co-occurring concepts in scientific papers. There, nodes represent scientific concepts, specifically biomolecules, and are linked when a paper mentions both in its title or abstract. This evolving network captures the field’s history and, using supercomputer simulations, provides insights into scientists’ collective behaviour and suggests more efficient research strategies 13 . Although creating semantic networks from concept co-occurrences extracts only a small amount of knowledge from each paper, it captures non-trivial and actionable content when applied to large datasets 2 , 4 , 13 , 14 , 15 . PaperRobot extends this approach by predicting new links from large medical knowledge graphs and formulating new ideas in human language as paper drafts 16 .

This approach was applied and extended to quantum physics 17 by building a semantic network of over 6,000 concepts. There, the authors (including one of us) formulated the prediction of new research trends and connections as an ML task, with the goal of identifying concept pairs not yet jointly discussed in the literature but likely to be investigated in the future. This prediction task was one component for personalized suggestions of new research ideas.

Link prediction in semantic networks

We formulate the prediction of future research topics as a link-prediction task in an exponentially growing semantic network in the AI field. The goal is to predict which unconnected nodes, representing scientific concepts not yet jointly researched, will be connected in the future.

Link prediction is a common problem in computer science, addressed with classical metrics and features, as well as ML techniques. Network theory-based methods include local motif-based approaches 18 , 19 , 20 , 21 , 22 , linear optimization 23 , global perturbations 24 and stochastic block models 25 . ML works optimized a combination of predictors 26 , with further discussion in a recent review 27 .

In ref. 17 , 17 hand-crafted features were used for this task. In the Science4Cast competition, the goal was to find more precise methods for link-prediction tasks in semantic networks (a semantic network of AI that is ten times larger than the one in ref. 17 ).

Potential for idea generation in science

The long-term goal of predictions and suggestions in semantic networks is to provide new ideas to individual researchers. In a way, we hope to build a creative artificial muse in science 28 . We can bias or constrain the model to give topic suggestions that are related to the research interest of individual scientists, or a pair of scientists to suggest topics for collaborations in an interdisciplinary setting.

Generation and analysis of the dataset

Dataset construction.

We create a dynamic semantic network using papers published on arXiv from 1992 to 2020 in the categories cs.AI, cs.LG, cs.NE and stat.ML. The 64,719 nodes represent AI concepts extracted from 143,000 paper titles and abstracts using Rapid Automatic Keyword Extraction (RAKE) and normalized via natural language processing (NLP) techniques and custom methods 29 . Although high-quality taxonomies such as the Computer Science Ontology (CSO) exist 30 , 31 , we choose not to use them for two reasons: the rapid growth of AI and ML may result in new concepts not yet in the CSO, and not all scientific domains have high-quality taxonomies like CSO. Our goal is to build a scalable approach applicable to any domain of science. However, future research could investigate merging these approaches (see ‘Extensions and future work’).

Concepts form the nodes of the semantic network, and edges are drawn when concepts co-appear in a paper title or abstract. Edges have time stamps based on the paper’s publication date, and multiple time-stamped edges between concepts are common. The network is edge-weighted, and the weight of an edge stands for the number of papers that connect two concepts. In total, this creates a time-evolving semantic network, depicted in Fig. 2 .

Utilizing 143,000 AI and ML papers on arXiv from 1992 to 2020, we create a list of concepts using RAKE and other NLP tools, which form nodes in a semantic network. Edges connect concepts that co-occur in titles or abstracts, resulting in an evolving network that expands as more concepts are jointly investigated. The task involves predicting which unconnected nodes (concepts not yet studied together) will connect within a few years. We present ten diverse statistical and ML methods to address this challenge.

Network-theoretical analysis

The published semantic network has 64,719 nodes and 17,892,352 unique undirected edges, with a mean node degree of 553. Many hub nodes greatly exceed this mean degree, as shown in Fig. 3 , For example, the highest node degrees are 466,319 (neural network), 198,050 (deep learning), 195,345 (machine learning), 169,555 (convolutional neural network), 159,403 (real world), 150,227 (experimental result), 127,642 (deep neural network) and 115,334 (large scale). We fit a power-law curve to the degree distribution p ( k ) using ref. 32 and obtained p ( k )  ∝   k −2.28 for degree k  ≥ 1,672. However, real complex network degree distributions often follow power laws with exponential cut-offs 33 . Recent work 34 has indicated that lognormal distributions fit most real-world networks better than power laws. Likelihood ratio tests from ref. 32 suggest truncated power law ( P  = 0.0031), lognormal ( P  = 0.0045) and lognormal positive ( P  = 0.015) fit better than power law, while exponential ( P  = 3 × 10 −10 ) and stretched exponential ( P  = 6 × 10 −5 ) are worse. We couldn’t conclusively determine the best fit with P  ≤ 0.1.

Nodes with the highest (466,319) and lowest (2) non-zero degrees are neural network and video compression technique, respectively. The most frequent non-zero degree is 64 (which occures 313 times). The plot, in log scale, omits 1,247 nodes with zero degrees.

We observe changes in network connectivity over time. Although degree distributions remained heavy-tailed, the ordering of nodes within the tail changed due to popularity trends. The most connected nodes and the years they became so include decision tree (1994), machine learning (1996), logic program (2000), neural network (2005), experimental result (2011), machine learning (2013, for a second time) and neural network (2015).

Connected component analysis in Fig. 4 reveals that the network grew more connected over time, with the largest group expanding and the number of connected components decreasing. Mid-sized connected components’ trajectories may expose trends, like image processing. A connected component with four nodes appeared in 1999 (brightness change, planar curve, local feature, differential invariant), and three more joined in 2000 (similarity transformation, template matching, invariant representation). In 2006, a paper discussing support vector machine and local feature merged this mid-sized group with the largest connected component.

Primary (left, blue) vertical axis: number of connected components with more than one node. Secondary (right, orange) vertical axis: number of nodes in the largest connected component. For example, the network in 2019 comprises of one large connected component with 63,472 nodes and 1,247 isolated nodes, that is, nodes with no edges. However, the 2001 network has 19 connected components with size greater than one, the largest of which has 2,733 nodes.

The semantic network reveals increasing centralization over time, with a smaller percentage of nodes (concepts) contributing to a larger fraction of edges (concept combinations). Figure 5 shows that the fraction of edges for high-degree nodes rises, while it decreases for low-degree nodes. The decreasing average clustering coefficient over time supports this trend, suggesting nodes are more likely to connect to high-degree central nodes. This could be due to the AI community’s focus on a few dominating methods or more consistent terminology use.

This cumulative histogram illustrates the fraction of nodes (concepts) corresponding to the fraction of edges (connections) for given years (1999, 2003, 2007, 2011, 2015 and 2019). The graph was generated by adding edges and nodes dated before each year. Nodes are sorted by increasing degrees. The y value at x  = 80 represents the fraction of edges contributed by all nodes in and below the 80th percentile of degrees.

Problem formulation

At the big picture, we aim to make predictions in an exponentially growing semantic network. The specific task involves predicting which two nodes v 1 and v 2 with degrees d ( v 1/ 2 ) ≥  c lacking an edge in the year (2021 −  δ ) will have w edges in 2021. We use δ  = 1, 3, 5, c  = 0, 5, 25 and w  = 1, 3, where c is a minimal degree. Note that c  = 0 is an intriguing special case where the nodes may not have an associated edge in the initial year, requiring the model to predict which nodes will connect to entirely new edges. The task w  = 3 goes beyond simple link prediction and seeks to identify uninvestigated concept pairs that will appear together in at least three papers. An interesting alternative task could be predicting the fastest-growing links, denoted as ‘trend’ prediction.

In this task, we provide a list of 10 million unconnected node pairs (each node having a degree ≥ c ) for the year (2021 −  δ ), with the goal of sorting this list by descending probability that they will have at least w edges in 2021.

For evaluation, we employ the receiver operating characteristic (ROC) curve 35 , which plots the true-positive rate against the false-positive rate at various threshold settings. We use the area under the curve (AUC) of the ROC curve as our evaluation metric. The advantage of AUC over mean square error is its independence from the data distribution. Specifically, in our case, where the two classes have a highly asymmetric distribution (with only about 1–3% of newly connected edges) and the distribution changes over time, AUC offers meaningful interpretation. Perfect predictions yield AUC = 1, whereas random predictions result in AUC = 0.5. AUC represents the percentage that a random true element is ranked higher than a random false one. For other metrics, see ref. 36 .

To tackle this task, models can use the complete information of the semantic network from the year (2021 −  δ ) in any way possible. In our case, all presented models generate a dataset for learning to make predictions from (2021 − 2 δ ) to (2021 −  δ ). Once the models successfully complete this task, they are applied to the test dataset to make predictions from (2021 −  δ ) to 2021. All reported AUCs are based on the test dataset. Note that solving the test dataset is especially challenging due to the δ -year shift, causing systematic changes such as the number of papers and density of the semantic network.

AI-based solutions

We demonstrate various methods to predict new links in a semantic network, ranging from pure statistical approaches and neural networks with hand-crafted features (NF) to ML models without NF. The results are shown in Fig. 6 , with the highest AUC scores achieved by methods using NF as ML model inputs. Pure network features without ML are competitive, while pure ML methods have yet to outperform those with NF. Predicting links generated at least three times can achieve a quasi-deterministic AUC > 99.5%, suggesting an interesting target for computational sociology and science of science research. We have performed numerous tests to exclude data leakage in the benchmark dataset, overfitting or data duplication both in the set of articles and the set of concepts. We rank methods based on their performance, with model M1 as the best performing and model M8 as the least effective (for the prediction of a new edge with δ  = 3, c  = 0). Models M4 and M7 are subdivided into M4A, M4B, M7A and M7B, differing in their focus on feature or embedding selection (more details in Methods ).

Here we show the AUC values for different models that use machine learning techniques (ML), hand-crafted network features (NF) or a combination thereof. The left plot shows results for the prediction of a single new link (that is, w  = 1) and the right plot shows the results for the prediction of new triple links w  = 3. The task is to predict δ  = [1, 3, 5] years into the future, with cut-off values c  = [0, 5, 25]. We sort the models by the the results for the task ( w  = 1,  δ  = 3,  c  = 0), which was the task in the Science4Cast competition. Data points that are not shown have a AUC below 0.6 or are not computed due to computational costs. All AUC values reported are computed on a validation dataset δ years ahead of the training dataset that the models have never seen. Note that the prediction of new triple edges can be performed nearly deterministic. It will be interesting to understand the origin of this quasi-deterministic pattern in AI research, for example, by connecting it to the research interests of scientists 88 .

Model M1: NF + ML. This approach combines tree-based gradient boosting with graph neural networks, using extensive feature engineering to capture node centralities, proximity and temporal evolution 37 . The Light Gradient Boosting Machine (LightGBM) model 38 is employed with heavy regularization to combat overfitting due to the scarcity of positive examples, while a time-aware graph neural network learns dynamic node representations.

Model M2: NF + ML. This method utilizes node and edge features (as well as their first and second derivatives) to predict link formation probabilities 39 . Node features capture popularity, and edge features measure similarity. A multilayer perceptron with rectified linear unit (ReLU) activation is used for learning. Cold start issues are addressed with feature imputation.

Model M3: NF + ML. This method captures hand-crafted node features over multiple time snapshots and employs a long short-term memory (LSTM) to learn time dependencies 40 . The features were selected to be highly informative while having a low computational cost. The final configuration uses degree centrality, degree of neighbours and common neighbours as features. The LSTM outperforms fully connected neural networks.

Model M4: pure NF. Two purely statistical methods, preferential attachment 41 and common neighbours 27 , are used 42 . Preferential attachment is based on node degrees, while common neighbours relies on the number of shared neighbours. Both methods are computationally inexpensive and perform competitively with some learning-based models.

Model M5: NF + ML. Here, ten groups of first-order graph features are extracted to obtain neighbourhood and similarity properties, with principal component analysis 43 applied for dimensionality reduction 44 . A random forest classifier is trained on the balanced dataset to predict new links.

Model M6: NF + ML. The baseline solution uses 15 hand-crafted features as input to a four-layer neural network, predicting the probability of link formation between node pairs 17 .

Model M7: end-to-end ML (auto node embedding). The baseline solution is modified to use node2vec 45 and ProNE embeddings 46 instead of hand-crafted features. The embeddings are input to a neural network with two hidden layers for link prediction.

Model M8: end-to-end ML (transformers). This method learns features in an unsupervised manner using transformers 47 . Node2vec embeddings 45 , 48 are generated for various snapshots of the adjacency matrix, and a transformer model 49 is pre-trained as a feature extractor. A two-layer ReLU network is used for classification.

Extensions and future work

Developing an AI that suggests research topics to scientists is a complex task, and our link-prediction approach in temporal networks is just the beginning. We highlight key extensions and future work directly related to the ultimate goal of AI for AI.

High-quality predictions without feature engineering. Interestingly, the most effective methods utilized carefully crafted features on a graph with extracted concepts as nodes and edges representing their joint publication history. Investigating whether end-to-end deep learning can solve tasks without feature engineering will be a valuable next step.

Fully automated concept extraction. Current concept lists, generated by RAKE’s statistical text analysis, demand time-consuming code development to address irrelevant term extraction (for example, verbs, adjectives). A fully automated NLP technique that accurately extracts meaningful concepts without manual code intervention would greatly enhance the process.

Leveraging ontology taxonomies. Alongside fully automated concept extraction, utilizing established taxonomies such as the CSO 30 , 31 , Wikipedia-extracted concepts, book indices 17 or PhySH key phrases is crucial. Although not comprehensive for all domains, these curated datasets often contain hierarchical and relational concept information, greatly improving prediction tasks.

Incorporating relation extraction. Future work could explore relation extraction techniques for constructing more accurate, sparser semantic networks. By discerning and classifying meaningful concept relationships in abstracts 50 , 51 , a refined AI literature representation is attainable. Using NLP tools for entity recognition, relationship identification and classification, this approach may enhance prediction performance and novel research direction identification.

Generation of new concepts. Our work predicts links between known concepts, but generating new concepts using AI remains a challenge. This unsupervised task, as explored in refs. 52 , 53 , involves detecting concept clusters with dynamics that signal new concept formation. Incorporating emerging concepts into the current framework for suggesting research topics is an intriguing future direction.

Semantic information beyond concept pairs. Currently, abstracts and titles are compressed into concept pairs, but more comprehensive information extraction could yield meaningful predictions. Exploring complex data structures such as hypergraphs 54 may be computationally demanding, but clever tricks could reduce complexity, as shown in ref. 55 . Investigating sociological factors or drawing inspiration from material science approaches 56 may also improve prediction tasks. A recent dataset for the study of the science of science also includes more complex data structures than the ones used in our paper, including data from social networks such as Twitter 57 .

Predictions of scientific success. While predicting new links between concepts is valuable, assessing their potential impact is essential for high-quality suggestions. Introducing a metric of success, like estimated citation numbers or citation growth rate, can help gauge the importance of these connections. Adapting citation prediction techniques from the science of science 58 , 59 , 60 , 61 to semantic networks offers a promising research direction.

Anomaly detections. Predicting likely connections may not align with finding surprising research directions. One method for identifying surprising suggestions involves constraining cosine similarity between vertices 62 , which measures shared neighbours and can be associated with semantic (dis)similarity. Another approach is detecting anomalies in semantic networks, which are potential links with extreme properties 63 , 64 . While scientists often focus on familiar topics 3 , 4 , greater impact results from unexpected combinations of distant domains 12 , encouraging the search for surprising associations.

End-to-end formulation. Our method breaks down the goal of extracting knowledge from scientific literature into subtasks, contrasting with end-to-end deep learning that tackles problems directly without subproblems 65 , 66 . End-to-end approaches have shown great success in various domains 67 , 68 , 69 . Investigating whether such an end-to-end solution can achieve similar success in our context would be intriguing.

Our method represents a crucial step towards developing a tool that can assist scientists in uncovering novel avenues for exploration. We are confident that our outlined ideas and extensions pave the way for achieving practical, personalized, interdisciplinary AI-based suggestions for new impactful discoveries. We firmly believe that such a tool holds the potential to become a influential catalyst, transforming the way scientists approach research questions and collaborate in their respective fields.

Details on concept set generation and application

In this section, we provide details on the generation of our list of 64,719 concepts. For more information, the code is accessible on GitHub . The entire approach is designed for immediate scalability to other domains.

Initially, we utilized approximately 143,000 arXiv papers from the categories cs.AI, cs.LG, cs.NE and stat.ML spanning 1992 to 2020. The omission of earlier data has a negligible effect on our research question, as we show below. We then iterated over each individual article, employing RAKE (with an extended stopword list) to suggest concept candidates, which were subsequently stored.

Following the iteration, we retained concepts composed of at least two words (for example, neural network) appearing in six or more articles, as well as concepts comprising a minimum of three words (for example, recurrent neural network) appearing in three or more articles. This initial filter substantially reduced noise generated by RAKE, resulting in a list of 104,948 concepts.

Lastly, we developed an automated filtering tool to further enhance the quality of the concept list. This tool identified common, domain-independent errors made by RAKE, which primarily included phrases that were not concepts (for example, dataset provided or discuss open challenge). We compiled a list of 543 words not part of meaningful concepts, including verbs, ordinal numbers, conjunctions and adverbials. Ultimately, this process produced our final list of 64,719 concepts employed in our study. No further semantic concept/entity linking is applied.

By this construction, the test sets with c  = 0 could lead to very rare contamination of the dataset. That is because each concept will have at least one edge in the final dataset. The effects, however, are negligible.

The distribution of concepts in the articles can be seen in Extended Data Fig. 1 . As an example, we show the extraction of concepts from five randomly chosen papers:

Memristor hardware-friendly reinforcement learning 70 : ‘actor critic algorithm’, ‘neuromorphic hardware implementation’, ‘hardware neural network’, ‘neuromorphic hardware system’, ‘neural network’, ‘large number’, ‘reinforcement learning’, ‘case study’, ‘pre training’, ‘training procedure’, ‘complex task’, ‘high performance’, ‘classical problem’, ‘hardware implementation’, ‘synaptic weight’, ‘energy efficient’, ‘neuromorphic hardware’, ‘control theory’, ‘weight update’, ‘training technique’, ‘actor critic’, ‘nervous system’, ‘inverted pendulum’, ‘explicit supervision’, ‘hardware friendly’, ‘neuromorphic architecture’, ‘hardware system’.

Automated deep learning analysis of angiography video sequences for coronary artery disease 71 : ‘deep learning approach’, ‘coronary artery disease’, ‘deep learning analysis’, ‘traditional image processing’, ‘deep learning’, ‘image processing’, ‘f1 score’, ‘video sequence’, ‘error rate’, ‘automated analysis’, ‘coronary artery’, ‘vessel segmentation’, ‘key frame’, ‘visual assessment’, ‘analysis method’, ‘analysis pipeline’, ‘coronary angiography’, ‘geometrical analysis’.

Demographic influences on contemporary art with unsupervised style embeddings 72 : ‘classification task’, ‘social network’, ‘data source’, ‘visual content’, ‘graph network’, ‘demographic information’, ‘social connection’, ‘visual style’, ‘historical dataset’, ‘novel information’

The utility of general domain transfer learning for medical language tasks 73 : ‘natural language processing’, ‘long short term memory’, ‘logistic regression model’, ‘transfer learning technique’, ‘short term memory’, ‘average f1 score’, ‘class classification model’, ‘domain transfer learning’, ‘weighted average f1 score’, ‘medical natural language processing’, ‘natural language process’, ‘transfer learning’, ‘f1 score’, ’natural language’, ’deep model’, ’logistic regression’, ’model performance’, ’classification model’, ’text classification’, ’regression model’, ’nlp task’, ‘short term’, ‘medical domain’, ‘weighted average’, ‘class classification’, ‘bert model’, ‘language processing’, ‘biomedical domain’, ‘domain transfer’, ‘nlp model’, ‘main model’, ‘general domain’, ‘domain model’, ‘medical text’.

Fast neural architecture construction using envelopenets 74 : ‘neural network architecture’, ‘neural architecture search’, ‘deep network architecture’, ‘image classification problem’, ‘neural architecture search method’, ‘neural network’, ‘reinforcement learning’, ‘deep network’, ‘image classification’, ‘objective function’, ‘network architecture’, ‘classification problem’, ‘evolutionary algorithm’, ‘neural architecture’, ‘base network’, ‘architecture search’, ‘training epoch’, ‘search method’, ‘image class’, ‘full training’, ‘automated search’, ‘generated network’, ‘constructed network’, ‘gpu day’.

Time gap between the generation of edges

We use articles from arXiv, which only goes back to the year 1992. However, of course, the field of AI exists at least since the 1960s 75 . Thus, this raises the question whether the omission of the first 30–40 years of research has a crucial impact in the prediction task we formulate, specifically, whether edges that we consider as new might not be so new after all. Thus, in Extended Data Fig. 2 , we compute the time between the formation of edges between the same concepts, taking into account all or just the first edge. We see that the vast majority of edges are formed within short time periods, thus the effect of omission of early publication has a negligible effect for our question. Of course, different questions might be crucially impacted by the early data; thus, a careful choice of the data source is crucial 61 .

Positive examples in the test dataset

Table 1 shows the number of positive cases within the 10 million examples in the 18 test datasets that are used for evaluation.

Publication rates in quantum physics

Another field of research that gained a lot of attention in the recent years is quantum physics. This field is also a strong adopter of arXiv. Thus, we analyse in the same way as for AI in Fig. 1 . We find in Extended Data Fig. 3 no obvious exponential increase in papers per month. A detailed analysis of other domains is beyond the current scope. It will be interesting to investigate the growth rates in different scientific disciplines in more detail, especially given that exponential increase has been observed in several aspects of the science of science 3 , 76 .

Details on models M1–M8

What follows are more detailed explanations of the models presented in the main text. All codes are available at GitHub. The feature importance of the best model M1 is shown here, those of other models are analysed in the respective workshop contributions (cited in the subsections).

Details on M1

The best-performing solution is based on a blend of a tree-based gradient boosting approach and a graph neural network approach 37 . Extensive feature engineering was conducted to capture the centralities of the nodes, the proximity between node pairs and their evolution over time. The centrality of a node is captured by the number of neighbours and the PageRank score 77 , while the proximity between a node pair is derived using the Jaccard index. We refer the reader to ref. 37 for the list of all features and their feature importance.

The tree-based gradient boosting approach uses LightGBM 38 and applies heavy regularization to combat overfitting due to the scarcity of positive samples. The graph neural network approach employs a time-aware graph neural network to learn node representations on dynamic semantic networks. The feature importance of model M1, averaged over 18 datasets, is shown in Table 2 . It shows that the temporal features do contribute largely to the model performance, but the model remains strong even when they are removed. An example of the evolution of the training (from 2016 to 2019) and test set (2019 to 2021) for δ  = 3, c  = 25, ω  = 1 is shown in Extended Data Fig. 4 .

Details on M2

The second method assumes that the probability that nodes u and v form an edge in the future is a function of the node features f ( u ), f ( v ) and some edge feature h ( u ,  v ). We chose node features f that capture popularity at the current time t 0 (such as degree, clustering coefficient 78 , 79 and PageRank 77 ). We also use these features’ first and second time derivatives to capture the evolution of the node’s popularity over time. After variable selection during training, we chose h to consist of the HOP-rec score (high-order proximity for implicit recommendation) 80 , 81 and a variation of the Dice similarity score 82 as a measure of similarity between nodes. In summary, we use 31 node features for each node, and two edge features, which gives 31 × 2 + 2 = 64 features in total. These features are then fed into a small multilayer perceptron (5 layers, each with 13 neurons) with ReLU activation.

Cold start is the problem that some nodes in the test set do not appear in the training set. Our strategy for a cold start is imputation. We say a node v is seen if it appeared in the training data, and unseen otherwise; similarly, we say that a node is born at time t if t is the first time stamp where an edge linking this node has appeared. The idea is that an unseen node is simply a node born in the future, so its features should look like a recently born node in the training set. If a node is unseen, then we impute its features as the average of the features of the nodes born recently. We found that with imputation during training, the test AUC scores across all models consistently increased by about 0.02. For a complete description of this method, we refer the reader to ref. 39 .

Details on M3

This approach, detailed in ref. 40 , uses hand-crafted node features that have been captured in multiple time snapshots (for example, every year) and then uses an LSTM to benefit from learning the time dependencies of these features. The final configuration uses two main types of feature: node features including degree and degree of neighbours, and edge features including common neighbours. In addition, to balance the training data, the same number of positive and negative instances have been randomly sampled and combined.

One of the goals was to identify features that are very informative with a very low computational cost. We found that the degree centrality of the nodes is the most important feature, and the degree centrality of the neighbouring nodes and the degree of mutual neighbours gave us the best trade-off. As all of the extracted features’ distributions are highly skewed to the right, meaning most of the features take near zero values, using a power transform such as Yeo–Johnson 83 helps to make the distributions more Gaussian, which boosts the learning. Finally, for the link-prediction task, we saw that LSTMs perform better than fully connected neural networks.

Details on M4

The following two methods are based on a purely statistical analysis of the test data and are explained in detail in ref. 42 .

Preferential attachment. In the network analysis, we concluded that the growth of this dataset tends to maintain a heavy-tailed degree distribution, often associated with scale-free networks. As mentioned before the γ value of the degree distribution is very close to 2, suggesting that preferential attachment 41 is probably the main organizational principle of the network. As such, we implemented a simple prediction model following this procedure. Preferential attachment scores in link prediction are often quantified as

with k i , j the degree of nodes i and j . However, this assumes the scoring of links between nodes that are already connected to the network, that is k i , j  > 0, which is not the case for all the links we must score in the dataset. As a result, we define our preferential attachment model as

Using this simple model with no free parameters we could score new links and compare them with the other models. Immediately we note that preferential attachment outperforms some learning-based models, even if it never manages to reach the top AUC, but it is extremely simple and with negligible computational cost.

Common neighbours. We explore another network-based approach to score the links. Indeed, while the preferential attachment model we derived performed well, it uses no information about the distance between i and j , which is a popular feature used in link-prediction methods 27 . As such, we decided to test a method known as common neighbours 18 . We define Γ ( i ) as the set of neighbors of node i and Γ ( i ) ∩  Γ ( j ) as the set of common neighbours between nodes i and j . We can easily score the nodes with

the intuition being that nodes that share a larger number of neighbours are more likely to be connected than distant nodes that do not share any.

Evaluating this score for each pair ( i ,  j ) on the dataset of unconnected pairs, which can be computed as the second power of the adjacency matrix, A 2 , we obtained an AUC that is sometimes higher than preferential attachment and sometimes lower than it but is still consistently quite close with the best learning-based models.

Details on M5

This method is based on ref. 44 . First, ten groups of first-order graph features are extracted to get some neighbourhood and similarity properties from each pair of nodes: degree centrality of nodes, pair’s total number of neighbours, common neighbours index, Jaccard coefficient, Simpson coefficient, geometric coefficient, cosine coefficient, Adamic–Adar index, resource allocation index and preferential attachment index. They are obtained for three consecutive years to capture the temporal dynamics of the semantic network, leading to a total of 33 features. Second, principal component analysis 43 is applied to reduce the correlation between features, speed up the learning process and improve generalization, which results in a final set of seven latent variables. Lastly, a random forest classifier is trained (using a balanced dataset) to estimate the likelihood of new links between the AI concepts.

In this paper, a modification was performed in relation to the original formulation of the method 44 : two of the original features, average neighbour degree and clustering coefficient, were infeasible to extract for some of the tasks covered in this paper, as their computation can be heavy for such a very large network, and they were discarded. Due to some computational memory issues, it was not possible to run the model for some of the tasks covered in this study, and so those results are missing.

Details on M6

The baseline solution for the Science4Cast competition was closely related to the model presented in ref. 17 . It uses 15 hand-crafted features of a pair of nodes v 1 and v 2 . (Degrees of v 1 and v 2 in the current year and previous two years are six properties. The number of shared neighbours in total of v 1 and v 2 in the current year and previous two years are six properties. The number of shared neighbours between v 1 and v 2 in the current year and the previous two years are three properties). These 15 features are the input of a neural network with four layers (15, 100, 10 and 1 neurons), intending to predict whether the nodes v 1 and v 2 will have w edges in the future. After the training, the model computes the probability for all 10 million evaluation examples. This list is sorted and the AUC is computed.

Details on M7

The solution M7 was not part of the Science4Cast competition and therefore not described in the corresponding proceedings, thus we want to add more details.

The most immediate way one can apply ML to this problem is by automating the detection of features. Quite simply, the baseline solution M6 is modified such that instead of 15 hand-crafted features, the neural network is instead trained on features extracted from a graph embedding. We use two different embedding approaches. The first method is employs node2vec (M7A) 45 , for which we use the implementations provided in the nodevectors Python package 84 . The second one uses the ProNE embedding (M7B) 46 , which is based on sparse matrix factorizations modulated by the higher-order Cheeger inequality 85 .

The embeddings generate a 32-dimensional representation for each node, resulting in edge representations in [0, 1] 64 . These features are input into a neural network with two hidden layers of size 1,000 and 30. Like M6, the model computes the probability for evaluation examples to determine the ROC. We compare ProNE to node2vec, a common graph embedding method using a biased random walk procedure with return and in–out parameters, which greatly affect network encoding. Initial experiments used default values for a 64-dimensional encoding before inputting into the neural network. The higher variance in node2vec predictions is probably due to its sensitivity to hyperparameters. While ProNE is better suited for general multi-dataset link prediction, node2vec’s sensitivity may help identify crucial network features for predicting temporal evolution.

Details on M8

This model, which is detailed in ref. 47 , does not use any hand-crafted features but learns them in a completely unsupervised manner. To do so, we extract various snapshots of the adjacency matrix through time, capturing graphs in the form of A t for t  = 1994, …, 2019. We then embed each of these graphs into 128-dimensional Euclidean space via node2vec 45 , 48 . For each node u in the semantic graph, we extract different 128-dimensional vector embeddings n u ( A 1994 ), …,  n u ( A 2019 ).

Transformers have performed extremely well in NLP tasks 49 ; thus, we apply them to learn the dynamics of the embedding vectors. We pre-train a transformer to help classify node pairs. For the transformer, the encoder and decoder had 6 layers each; we used 128 as the embedding dimension, 2,048 as the feed-forward dimension and 8-headed attention. This transformer acts as our feature extractor. Once we pre-train our transformer, we add a two-layer ReLU network with hidden dimension 128 as a classifier on top.

Data availability

All 18 datasets tested in this paper are available via Zenodo at https://doi.org/10.5281/zenodo.7882892 ref. 86 .

Code availability

All of the models and codes described above can be found via GitHub at https://github.com/artificial-scientist-lab/FutureOfAIviaAI ref. 5 and a permanent Zenodo record at https://zenodo.org/record/8329701 ref. 87 .

Clauset, A., Larremore, D. B. & Sinatra, R. Data-driven predictions in the science of science. Science 355 , 477–480 (2017).

Article   Google Scholar  

Evans, J. A. & Foster, J. G. Metaknowledge. Science 331 , 721–725 (2011).

Article   MathSciNet   MATH   Google Scholar  

Fortunato, S. et al. Science of science. Science 359 , eaao0185 (2018).

Wang, D. & Barabási, A.-L. The Science of Science (Cambridge Univ. Press, 2021).

Krenn, M. et al. FutureOfAIviaAI. GitHub https://github.com/artificial-scientist-lab/FutureOfAIviaAI (2023).

Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33 , 1877–1901 (2020).

Google Scholar  

Rae, J. W. et al. Scaling language models: methods, analysis & insights from training gopher. Preprint at https://arxiv.org/abs/2112.11446 (2021).

Smith, S. et al. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B, a large-scale generative language model. Preprint at https://arxiv.org/abs/2201.11990 (2022).

Chowdhery, A. et al. Palm: scaling language modeling with pathways. Preprint at https://arxiv.org/abs/2204.02311 (2022).

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large language models are zero-shot reasoners. Preprint at https://arxiv.org/abs/2205.11916 (2022).

Zhang, H., Li, L. H., Meng, T., Chang, K.-W. & Broeck, G. V. d. On the paradox of learning to reason from data. Preprint at https://arxiv.org/abs/2205.11502 (2022).

Rzhetsky, A., Foster, J. G., Foster, I. T. & Evans, J. A. Choosing experiments to accelerate collective discovery. Proc. Natl Acad. Sci. USA 112 , 14569–14574 (2015).

Foster, J. G., Rzhetsky, A. & Evans, J. A. Tradition and innovation in scientists’ research strategies. Am. Sociol. Rev. 80 , 875–908 (2015).

Van Eck, N. J. & Waltman, L. Text mining and visualization using vosviewer. Preprint at https://arxiv.org/abs/1109.2058 (2011).

Van Eck, N. J. & Waltman, L. in Measuring Scholarly Impact: Methods and Practice (eds Ding, Y. et al.) 285–320 (Springer, 2014).

Wang, Q. et al. Paperrobot: Incremental draft generation of scientific ideas. Preprint at https://arxiv.org/abs/1905.07870 (2019).

Krenn, M. & Zeilinger, A. Predicting research trends with semantic and neural networks with an application in quantum physics. Proc. Natl Acad. Sci. USA 117 , 1910–1916 (2020).

Liben-Nowell, D. & Kleinberg, J. The link-prediction problem for social networks. J. Am. Soc. Inf. Sci. Technol. 58 , 1019–1031 (2007).

Albert, I. & Albert, R. Conserved network motifs allow protein–protein interaction prediction. Bioinformatics 20 , 3346–3352 (2004).

Zhou, T., Lü, L. & Zhang, Y.-C. Predicting missing links via local information. Eur. Phys. J. B 71 , 623–630 (2009).

Article   MATH   Google Scholar  

Kovács, I. A. et al. Network-based prediction of protein interactions. Nat. Commun. 10 , 1240 (2019).

Muscoloni, A., Abdelhamid, I. & Cannistraci, C. V. Local-community network automata modelling based on length-three-paths for prediction of complex network structures in protein interactomes, food webs and more. Preprint at bioRxiv https://doi.org/10.1101/346916 (2018).

Pech, R., Hao, D., Lee, Y.-L., Yuan, Y. & Zhou, T. Link prediction via linear optimization. Physica A 528 , 121319 (2019).

Lü, L., Pan, L., Zhou, T., Zhang, Y.-C. & Stanley, H. E. Toward link predictability of complex networks. Proc. Natl Acad. Sci. USA 112 , 2325–2330 (2015).

Guimerà, R. & Sales-Pardo, M. Missing and spurious interactions and the reconstruction of complex networks. Proc. Natl Acad. Sci. USA 106 , 22073–22078 (2009).

Ghasemian, A., Hosseinmardi, H., Galstyan, A., Airoldi, E. M. & Clauset, A. Stacking models for nearly optimal link prediction in complex networks. Proc. Natl Acad. Sci. USA 117 , 23393–23400 (2020).

Zhou, T. Progresses and challenges in link prediction. iScience 24 , 103217 (2021).

Krenn, M. et al. On scientific understanding with artificial intelligence. Nat. Rev. Phys. 4 , 761–769 (2022).

Rose, S., Engel, D., Cramer, N. & Cowley, W. in Text Mining: Applications and Theory (eds Berry, M. W. & Kogan, J.) Ch. 1 (Wiley, 2010).

Salatino, A. A., Thanapalasingam, T., Mannocci, A., Osborne, F. & Motta, E. The computer science ontology: a large-scale taxonomy of research areas. In Proc. Semantic Web–ISWC 2018: 17th International Semantic Web Conference Part II Vol. 17, 187–205 (Springer, 2018).

Salatino, A. A., Osborne, F., Thanapalasingam, T. & Motta, E. The CSO classifier: ontology-driven detection of research topics in scholarly articles. In Proc. Digital Libraries for Open Knowledge: 23rd International Conference on Theory and Practice of Digital Libraries Vol. 23, 296–311 (Springer, 2019).

Alstott, J., Bullmore, E. & Plenz, D. powerlaw: a Python package for analysis of heavy-tailed distributions. PLoS ONE 9 , e85777 (2014).

Fenner, T., Levene, M. & Loizou, G. A model for collaboration networks giving rise to a power-law distribution with an exponential cutoff. Soc. Netw. 29 , 70–80 (2007).

Broido, A. D. & Clauset, A. Scale-free networks are rare. Nat. Commun. 10 , 1017 (2019).

Fawcett, T. ROC graphs: notes and practical considerations for researchers. Pattern Recognit. Lett. 31 , 1–38 (2004).

Sun, Y., Wong, A. K. & Kamel, M. S. Classification of imbalanced data: a review. Int. J. Pattern Recognit. Artif. Intell. 23 , 687–719 (2009).

Lu, Y. Predicting research trends in artificial intelligence with gradient boosting decision trees and time-aware graph neural networks. In 2021 IEEE International Conference on Big Data (Big Data) 5809–5814 (IEEE, 2021).

Ke, G. et al. LightGBM: a highly efficient gradient boosting decision tree. In Proc. 31st International Conference on Neural Information Processing Systems 3149–3157 (Curran Associates Inc., 2017).

Tran, N. M. & Xie, Y. Improving random walk rankings with feature selection and imputation Science4Cast competition, team Hash Brown. In 2021 IEEE International Conference on Big Data (Big Data) 5824–5827 (IEEE, 2021).

Sanjabi, N. Efficiently predicting scientific trends using node centrality measures of a science semantic network. In 2021 IEEE International Conference on Big Data (Big Data) 5820–5823 (IEEE, 2021).

Barabási, A.-L. Network science. Phil. Trans. R. Soci. A 371 , 20120375 (2013).

Moutinho, J. P., Coutinho, B. & Buffoni, L. Network-based link prediction of scientific concepts—a Science4Cast competition entry. In 2021 IEEE International Conference on Big Data (Big Data) 5815–5819 (IEEE, 2021).

Jolliffe, I. T. & Cadima, J. Principal component analysis: a review and recent developments. Phil. Trans. R. Soc. A 374 , 20150202 (2016).

Valente, F. Link prediction of artificial intelligence concepts using low computational power. In 2021 IEEE International Conference on Big Data (Big Data) 5828–5832 (2021).

Grover, A. & Leskovec, J. node2vec: scalable feature learning for networks. In Proc. 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 855–864 (ACM, 2016).

Zhang, J., Dong, Y., Wang, Y., Tang, J. & Ding, M. ProNE: fast and scalable network representation learning. In Proc. Twenty-Eighth International Joint Conference on Artificial Intelligence 4278–4284 (International Joint Conferences on Artificial Intelligence Organization, 2019).

Lee, H., Sonthalia, R. & Foster, J. G. Dynamic embedding-based methods for link prediction in machine learning semantic network. In 2021 IEEE International Conference on Big Data (Big Data) 5801–5808 (IEEE, 2021).

Liu, R. & Krishnan, A. PecanPy: a fast, efficient and parallelized python implementation of node2vec. Bioinformatics 37 , 3377–3379 (2021).

Vaswani, A. et al. Attention is all you need. In Proc. 31st International Conference on Neural Information Processing Systems 6000–6010 (Curran Associates Inc., 2017).

Zelenko, D., Aone, C. & Richardella, A. Kernel methods for relation extraction. J. Mach. Learn. Res. 3 , 1083–1106 (2003).

MathSciNet   MATH   Google Scholar  

Bach, N. & Badaskar, S. A review of relation extraction. Literature Review for Language and Statistics II 2 , 1–15 (2007).

Salatino, A. A., Osborne, F. & Motta, E. How are topics born? Understanding the research dynamics preceding the emergence of new areas. PeerJ Comput. Sc. 3 , e119 (2017).

Salatino, A. A., Osborne, F. & Motta, E. AUGUR: forecasting the emergence of new research topics. In Proc. 18th ACM/IEEE on Joint Conference on Digital Libraries 303–312 (IEEE, 2018).

Battiston, F. et al. The physics of higher-order interactions in complex systems. Nat. Phys. 17 , 1093–1098 (2021).

Coutinho, B. C., Wu, A.-K., Zhou, H.-J. & Liu, Y.-Y. Covering problems and core percolations on hypergraphs. Phys. Rev. Lett. 124 , 248301 (2020).

Article   MathSciNet   Google Scholar  

Olivetti, E. A. et al. Data-driven materials research enabled by natural language processing and information extraction. Appl. Phys. Rev. 7 , 041317 (2020).

Lin, Z., Yin, Y., Liu, L. & Wang, D. SciSciNet: a large-scale open data lake for the science of science research. Sci. Data 10 , 315 (2023).

Azoulay, P. et al. Toward a more scientific science. Science 361 , 1194–1197 (2018).

Liu, H., Kou, H., Yan, C. & Qi, L. Link prediction in paper citation network to construct paper correlation graph. EURASIP J. Wirel. Commun. Netw. 2019 , 1–12 (2019).

Reisz, N. et al. Loss of sustainability in scientific work. New J. Phys. 24 , 053041 (2022).

Frank, M. R., Wang, D., Cebrian, M. & Rahwan, I. The evolution of citation graphs in artificial intelligence research. Nat. Mach. Intell. 1 , 79–85 (2019).

Newman, M. Networks (Oxford Univ. Press, 2018).

Kwon, D. et al. A survey of deep learning-based network anomaly detection. Cluster Comput. 22 , 949–961 (2019).

Pang, G., Shen, C., Cao, L. & Hengel, A. V. D. Deep learning for anomaly detection: a review. ACM Comput. Surv. 54 , 1–38 (2021).

Collobert, R. et al. Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12 , 2493–2537 (2011).

MATH   Google Scholar  

LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521 , 436–444 (2015).

Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Commun. ACM 60 , 84–90 (2017).

Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518 , 529–533 (2015).

Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529 , 484–489 (2016).

Wu, N., Vincent, A., Strukov, D. & Xie, Y. Memristor hardware-friendly reinforcement learning. Preprint at https://arxiv.org/abs/2001.06930 (2020).

Zhou, C. et al. Automated deep learning analysis of angiography video sequences for coronary artery disease. Preprint at https://arxiv.org/abs/2101.12505 (2021).

Huckle, N., Garcia, N. & Nakashima, Y. Demographic influences on contemporary art with unsupervised style embeddings. In Proc. Computer Vision–ECCV 2020 Workshops Part II Vol. 16, 126–142 (Springer, 2020).

Ranti, D. et al. The utility of general domain transfer learning for medical language tasks. Preprint at https://arxiv.org/abs/2002.06670 (2020).

Kamath, P., Singh, A. & Dutta, D. Fast neural architecture construction using envelopenets. Preprint at https://arxiv.org/abs/1803.06744 (2018).

Minsky, M. Steps toward artificial intelligence. Proc. IRE 49 , 8–30 (1961).

Bornmann, L., Haunschild, R. & Mutz, R. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanit. Soc. Sci. Commun. 8 , 224 (2021).

Brin, S. & Page, L. The anatomy of a large-scale hypertextual web search engine. Comput. Netw. ISDN Syst. 30 , 107–117 (1998).

Holland, P. W. & Leinhardt, S. Transitivity in structural models of small groups. Comp. Group Studies 2 , 107–124 (1971).

Watts, D. J. & Strogatz, S. H. Collective dynamics of ‘small-world’ networks. Nature 393 , 440–442 (1998).

Yang, J.-H., Chen, C.-M., Wang, C.-J. & Tsai, M.-F. HOP-rec: high-order proximity for implicit recommendation. In Proc. 12th ACM Conference on Recommender Systems 140–144 (2018).

Lin, B.-Y. OGB_collab_project. GitHub https://github.com/brucenccu/OGB_collab_project (2021).

Sorensen, T. A. A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on danish commons. Biol. Skar. 5 , 1–34 (1948).

Yeo, I.-K. & Johnson, R. A. A new family of power transformations to improve normality or symmetry. Biometrika 87 , 954–959 (2000).

Ranger, M. nodevectors. GitHub https://github.com/VHRanger/nodevectors (2021).

Bandeira, A. S., Singer, A. & Spielman, D. A. A Cheeger inequality for the graph connection Laplacian. SIAM J. Matrix Anal. Appl. 34 , 1611–1630 (2013).

Krenn, M. et al. Predicting the future of AI with AI. Zenodo https://doi.org/10.5281/zenodo.7882892 (2023).

Krenn, M. et al. FutureOfAIviaAI code. Zenodo https://zenodo.org/record/8329701 (2023).

Jia, T., Wang, D. & Szymanski, B. K. Quantifying patterns of research-interest evolution. Nat. Hum. Behav. 1 , 0078 (2017).

Download references

Acknowledgements

We thank IARAI Vienna and IEEE for supporting and hosting the IEEE BigData Competition Science4Cast. We are specifically grateful to D. Kreil, M. Neun, C. Eichenberger, M. Spanring, H. Martin, D. Geschke, D. Springer, P. Herruzo, M. McCutchan, A. Mihai, T. Furdui, G. Fratica, M. Vázquez, A. Gruca, J. Brandstetter and S. Hochreiter for helping to set up and successfully execute the competition and the corresponding workshop. We thank X. Gu for creating Fig. 2 , and M. Aghajohari and M. Sadegh Akhondzadeh for helpful comments on the paper. The work of H.L., R.S. and J.G.F. was supported by grant TWCF0333 from the Templeton World Charity Foundation. H.L. is additionally supported by NSF grant DMS-1952339. J.P.M. acknowledges the support of FCT (Portugal) through scholarship SFRH/BD/144151/2019. B.C. thanks the support from FCT/MCTES through national funds and when applicable co-funded EU funds under the project UIDB/50008/2020, and FCT through the project CEECINST/00117/2018/CP1495/CT0001. N.M.T. and Y.X. are supported by NSF grant DMS-2113468, the NSF IFML 2019844 award to the University of Texas at Austin, and the Good Systems Research Initiative, part of University of Texas at Austin Bridging Barriers.

Open access funding provided by Max Planck Society.

Author information

Authors and affiliations.

Max Planck Institute for the Science of Light (MPL), Erlangen, Germany

Mario Krenn

Instituto de Telecomunicações, Lisbon, Portugal

Lorenzo Buffoni, Bruno Coutinho & João P. Moutinho

University of Toronto, Toronto, Ontario, Canada

Sagi Eppel & Andrew Gritsevskiy

University of California Los Angeles, Los Angeles, CA, USA

Jacob Gates Foster, Harlin Lee & Rishi Sonthalia

Cavendish Laboratories, Cavendish, VT, USA

Andrew Gritsevskiy

Institute of Advanced Research in Artificial Intelligence (IARAI), Vienna, Austria

Andrew Gritsevskiy & Michael Kopp

Alpha 8 AI, Toronto, Ontario, Canada

Independent Researcher, Barcelona, Spain

Nima Sanjabi

University of Texas at Austin, Austin, TX, USA

Ngoc Mai Tran

Independent Researcher, Leiria, Portugal

Francisco Valente

University of Pennsylvania, Philadelphia, PA, USA

Yangxinyu Xie

University of California, San Diego, CA, USA

You can also search for this author in PubMed   Google Scholar

Contributions

M. Krenn and R.Y. initiated the research. M. Krenn and M. Kopp organized the Science4Cast competition. M. Krenn generated the datasets and initial codes. S.E. and H.L. analysed the network-theoretical properties of the semantic network. M. Krenn, L.B., B.C., J.G.F., A.G, H.L., Y.L, J.P.M, N.S., R.S., N.M.T, F.V., Y.X and M. Kopp provided codes for the ten models. M. Krenn wrote the paper with input from all co-authors.

Corresponding author

Correspondence to Mario Krenn .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Machine Intelligence thanks Alexander Belikov, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Mirko Pieropan, in collaboration with the Nature Machine Intelligence team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended data fig. 1.

Number of concepts per article.

Extended Data Fig. 2

Time Gap between the generation of edges. Here, left shows the time it takes to create a new edge between two vertices and right shows the time between the first and the second edge.

Extended Data Fig. 3

Publications in Quantum Physics.

Extended Data Fig. 4

Evolution of the AUC during training for Model M1.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Krenn, M., Buffoni, L., Coutinho, B. et al. Forecasting the future of artificial intelligence with machine learning-based link prediction in an exponentially growing knowledge network. Nat Mach Intell 5 , 1326–1335 (2023). https://doi.org/10.1038/s42256-023-00735-0

Download citation

Received : 21 January 2023

Accepted : 11 September 2023

Published : 16 October 2023

Issue Date : November 2023

DOI : https://doi.org/10.1038/s42256-023-00735-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

• Reference Manager
• Simple TEXT file

People also looked at

Perspective article, artificial intelligence and machine learning in sport research: an introduction for non-data scientists.

• Institute for Health and Sport, Victoria University, Melbourne, VIC, Australia

In the last two decades, artificial intelligence (AI) has transformed the way in which we consume and analyse sports. The role of AI in improving decision-making and forecasting in sports, amongst many other advantages, is rapidly expanding and gaining more attention in both the academic sector and the industry. Nonetheless, for many sports audiences, professionals and policy makers, who are not particularly au courant or experts in AI, the connexion between artificial intelligence and sports remains fuzzy. Likewise, for many, the motivations for adopting a machine learning (ML) paradigm in sports analytics are still either faint or unclear. In this perspective paper, we present a high-level, non-technical, overview of the machine learning paradigm that motivates its potential for enhancing sports (performance and business) analytics. We provide a summary of some relevant research literature on the areas in which artificial intelligence and machine learning have been applied to the sports industry and in sport research. Finally, we present some hypothetical scenarios of how AI and ML could shape the future of sports.

Introduction

It was in Moneyball ( Lewis, 2004 ), the famous success storey of the Major League Baseball team “Oakland Athletics,” that using in-game play statistics came under focus as a means to assemble an exceptional team. Despite Oakland Athletics' relatively small budget, the adoption of a rigorous data-driven approach to assemble a new team led to the playoffs in the year 2002. An economic evaluation of the Moneyball hypothesis ( Hakes and Sauer, 2006 ) describes how, at the time, a baseball hitters' salary was not truly explained by the contribution of a player's batting skills to winning games. Oakland Athletics gained a big advantage over their competitors by identifying and exploiting this information gap. It's been almost two decades since Moneyball principles, or SABRmetrics ( Lewis, 2004 ) was introduced to baseball. SABR stands for Society for American Baseball Research and SABRmetricians are those scientists who gather the in-game data and analyse it to answer questions that will lead to improving team performance. Since the success of the Oakland Athletics, most MLB teams started employing SABRmetricians. The ongoing and exponential increase of computer processing power has further accelerated the ability to analyse “big data,” and indeed, computers increasingly are taking charge of the deeper analysis of data sets, through means of artificial intelligence (AI). Likewise, the surge in high-quality data collection and data aggregation (accomplished by organisations like Baseball Savant/StatCast, ESPN and others) are key ingredients to the spike in the accuracy and breadth of analytics that was observed in the MLB in recent years.

The adoption of AI and statistical modelling in sports has become therefore more prominent in recent years as new technologies and research applications are impacting professional sports at various levels of sophistication. The wide applicability of machine learning algorithms, combined with increasing computing processing power as well as access to more and new sources of data in recent years, has made sports organisations hungry for new applications and strategies. The overriding aim is still to make them more competitive on and off the field–in athletic and business performance. The benefits of leveraging the power of AI can, in that regard, take different forms from optimising business or technical decision making to enhancing athlete/team performance but also increasing demand for attendance at sporting events, as well as promoting alternative entertainment formats of the sport.

We next list some areas where AI and machine learning (ML) have left their footprints in the world of sports ( Beal et al., 2019 ) and provide some examples of applications in each (some of the listed applications could overlap with one or more of the areas).

• Game activity/analytics: match outcome modelling, player/ball Tracking, match event (e.g., shot) classification, umpire assistance, sports betting .

• Talent identification and acquisition: player recruitment, player performance measurement, biomechanics .

• Training and coaching: assessment of team formation efficacy, tactical planning, player injury modelling .

• Fan and business focused: measurement of a player's economic value, modelling demand for event attendance, ticket pricing optimisation (variable and dynamic), wearable and sensor design, highlight packaging, virtual and augmented reality sport applications, etc .

The field of AI (particularly ML) offers new methodologies that have proven to be beneficial for tackling the above challenges. In this perspective paper we aim to provide sports business professionals and non-technical sports audiences, coaches, business leaders, policy makers and stakeholders with an overview of the range of AI approaches used to analyse sport performance and business centric problems. We also discuss perspectives on how AI could shape the future of sports in the next few years.

Research on AI and ML in Sports

In this section, we will not be reviewing examples of how AI has been applied to sports for a specific application, but rather, we will look at the intersection of AI and sports at a more abstract level, discussing some research that either surveyed or summarised the application of AI and ML in sports.

One of the earliest works discussing the potential applications of artificial intelligence in sports performance, and its positive impact on improving decision-making is by Lapham and Bartlett (1995) . The paper discusses how expert systems (i.e., a knowledge-based database used for reasoning) can be used for sports biomechanics purposes. Bartlett (2006) reviewed developments in the use of AI in sports biomechanics (e.g., throwing, shot putting, football kicking, …) to show that, at the time of writing, expert systems were marginally used in sports biomechanics despite being popular for “gait analysis” whereas Artificial Neural Networks were used for applications such as performance patterns in training and movement patterns of sports performers. An Artificial Neural Network (ANN) is a system that mimics the functionality of a human brain. ANNs are used to solve computational problems or estimate functions from a given data input, by imitating the way neurons are fired or activated in the human brain. Several (layers of) artificial neurons, known as perceptrons, are connected to perform computations which return an output as a function of the provided input ( Anderson, 1995 ).

Bartlett (2006) predicted that multi-layer ANNs will play a big role in sports technique analysis in the future. Indeed, as we discuss later, multi-layer ANNs, now commonly referred to as Deep Learning, have become one of the most popular techniques in sports related analytics. Last but not least Bartlett (2006) described the applications of Evolutionary Computation and hybrid systems in the optimization of sports techniques and skill learning. Further discussion around the applications of AI in sports biomechanics can be found in Ratiu et al. (2010) . McCabe and Trevathan (2008) discussed the use of artificial intelligence for prediction of sporting outcomes, showing how the behaviour of teams can be modelled in different sporting contests using multi-layer ANNs.

Between 2006 and 2010, machine learning algorithms, particularly ANNs were becoming more popular amongst computer scientists. This was aided by the impressive improvements in computer hardware, but also due to a shift in mindset in the AI community. Large volumes of data were made public amongst researchers and scientists (e.g., ImageNet a visual database delivered by Stanford University), and new open-source machine learning competitions were organised (such as Netflix Prize and Kaggle). It is these types of events that have shaped the adoption of AI and machine learning in many different fields of study from medicine to econometrics and sports, by facilitating access to training data and offering free open-source tools and frameworks for leveraging the power of AI. Note that, in addition to ANN, other machine learning techniques are utilised in such competitions, and sometimes these can be used in combination with one another. For instance, some of the techniques that went into the winning of the Netflix prize include singular value decomposition combined with restricted Boltzmann machines and gradient boosted decision trees.

Other examples discussing ANNs in sports include Novatchkov and Baca (2013) who discuss how ANNs can be used for understanding the quality of execution, assisting athletes and coaches, and training optimisation. However, the applications of AI to sports analytics go beyond the use of ANNs. For example, Fister et al. (2015 ) discussed how nature-inspired AI algorithms can be used to investigate unsolved research problems regarding safe and effective training plans. Their approach ( Fister et al., 2015 ) relies on the notion of artificial collective intelligence ( Chmait et al., 2016 ; Chmait, 2017 ) and the adaptability of algorithms to adapt to a changing environment. The authors show how such algorithms can be used to develop an artificial trainer to recommend athletes with an informed training strategy after taking into consideration various factors related to the athlete's physique and readiness. Other types of scientific methods that include Bayesian approaches have been applied to determining player abilities ( Whitaker et al., 2021 ) but also predicting match outcomes ( Yang and Swartz, 2004 ). Bayesian analysis and learning is an approach for building (statistical and inference) models by updating the probability for a hypothesis as more evidence or information becomes available by using Bayes' theorem ( Ghosh et al., 2007 ).

There are numerous research papers in which AI and ML is applied to sport, and it is not our aim to comprehensively discuss these works here 1 . However, we refer to a recent survey that elaborates on this topic. Beal et al. (2019) surveyed the applications of AI in team sports. The authors summarised existing academic work, in a range of sports, tackling issues such as match outcome modelling, in-game tactical decision making, player performance in fantasy sport games, and managing professional players' sport injuries. Work by Nadikattu (2020) presents, at an abstract level, discussions on how AI can be implemented in (American) sports from enhancing player performance, to assisting coaches to come up with the right formations and tactics, to developing automated video highlights of sports matches and supporting referees using computer vision applications.

We emphasise that the application of AI in sports is not limited to topics of sports performance, athlete talent identification or the technical analysis of the game. The (off the field) business side of sports organisations is rapidly shifting towards a data driven culture led by developing profiles of their fans and their consumer preferences. As fans call for superior content and entertainment, sport organisations must react by delivering a customised experience to their patrons. This is often achieved by the use of statistical modelling as well as other machine learning solutions, for example, to understand the value of players from an economic perspective. As shown in Chmait et al. (2020a) , investigating the relationship between the talent and success of athletes (to determine the existence of what is referred to as superstardom phenomenon or star power) is becoming an important angle to explore value created in sport. To provide an idea of the extent of such work, we note some sports in which the relationship between famous players/teams and their effect on audience attendance or sport consumption has been studied:

• In soccer ( Brandes et al., 2008 ; Jewell, 2017 ),

• In Major League Baseball ( Ormiston, 2014 ; Lewis and Yoon, 2016 )

• In the National Basketball Association ( Berri et al., 2004 ; Jane, 2016 )

• In tennis: superstar player effect in demand for tennis tournament attendance ( Chmait et al., 2020a ), the presence of a stardom effect in social media ( Chmait et al., 2020b ), player effect on German television audience demand for live broadcast tennis matches ( Konjer et al., 2017 )

• And similarly, in Cricket ( Paton and Cooke, 2005 ), Hockey ( Coates and Humphreys, 2012 ), and in the Australian Football League ( Lenten, 2012 ).

AI algorithms are being used in Formula 1 (F1) to improve the racing tactics of competing teams by analysing data from hundreds of sensors in the F1 car. Recent work by Piccinotti (2021) shows how artificial intelligence can provide F1 with automated ways for identifying tyre replacement strategies by modelling pit-stop timing and frequency as sequential decision-making problems.

Researchers from Tennis Australia and Victoria University devised a racket recommendation technique based on real HawkEye (computer vision system) data. An algorithm was used to recommend a selection of rackets based on movement, hitting pattern and style of the player with the aim to improve the player's performance ( Krause, 2019 ).

Accurate and fair judging of sophisticated skills in sports like gymnastics is a difficult task. Recently, a judging system was developed by Fujitsu Ltd. The system scores a routine based on the angles of a gymnast's joints. It uses AI to analyse 3D laser sensors that capture the gymnasts' movements ( Atiković et al., 2020 ).

Finally, it is important to note the exceptionally successful adoption of AI in board games like Chess, Checkers, Shogi and the Chinese game of GO, as well as virtual games (like Dota2 and StarCraft). In the last couple of decades, AI has delivered a staggering rise in performance in such areas to the point that machines (almost) constantly defeat human world champions. We refer to some notable solutions like Schaeffer et al. (2007) Checkers artificial algorithm, DeepBlue defeating Kasparov in Chess ( Campbell et al., 2002 ), AlphaGo Zero defeating Lee Sedol in Go ( Silver et al., 2017 ) (noting that AlphaZero is also unbeatable in chess) and Vinyals et al. (2019) AlphaStar in StarcraftII as well as superhuman AI for multiplayer poker ( Brown and Sandholm, 2019 ). Commonly, in these types of games or sports, AI algorithms rely on a Reinforcement Learning approach (which we will describe later) as well as using techniques like the Monte-Carlo Search Trees to explore the game and devise robust strategies to solve and play these games. Some of the recent testbeds used to evaluate AI agents and algorithms are discussed in Hernández-Orallo et al. (2017 ). For a broader investigation of AI in board and virtual/computer games refer to Risi and Preuss (2020) .

The rise of applying AI and ML is unstoppable and to that end, one might be wondering how AI an ML tools work and why are they different from traditional summary analytics. We touch upon these considerations in the next section.

The Machine Learning Paradigm

To understand why ML is used in a wide range of applications, we need to take a look into the difference between recent AI approaches to learning and traditional analytics approaches. At a higher conceptual level, one can describe old or traditional approaches to sports analytics, as starting off with some set of rules that constitute the problem definition, some data that is to be processed using a program/application which will then deliver answers to the given problem. In contrast, in a machine learning/predictive analytics paradigm, the way this process works is fundamentally different. For instance, in some approaches of the ML paradigm, one typically starts by feeding the program with answers and corresponding data to a specific problem, with an algorithm narrowing down the rules of the problem. These rules are later used for making predictions and they are evaluated or validated by testing their accuracy over new (unseen) data.

To that end, machine learning is an area of AI that is concerned with algorithms that learn from data by performing some form of inductive learning. In simple terms, ML prediction could be described as a function 2 from a set of inputs i 1 , i 2 , …, i n , to forecast an unknown value y , as follows f ( w 1 * i 1 , w 2 * i 2 , …, w n * i n ) = y , where w t is the weight of input t .

Different types or approaches of ML are used for different types of problems. Some of the most popular are supervised learning, unsupervised learning , and reinforcement learning :

• In supervised learning, we begin by observing and recording both inputs (the i 's) and outputs (the y 's) of a system, for a given period of time. This data (collection of correct examples of inputs and their corresponding outputs) is then analysed to derive the rules that underly the dynamics of the observed system, i.e., the rules that map a given input to its correct output.

• Unlike the above, in unsupervised learning, the correct examples or outputs from a given system are not available. The task of the algorithm is to discover (previously unnoticed) patterns in the input data.

• In reinforcement learning, an algorithm (usually referred to as an agent) is designed to take a series of actions that maximise its cumulative payoff or rewards over time. The agent then builds a policy (a map of action selection rules) that return a probability of taking a given action under different conditions of the problem.

For a thorough introduction to the fundamentals of machine learning and the popular ML algorithms see Bonaccorso (2017) . The majority of AI applications in sports are based on one or more of the above approaches to ML. In fact, in most predictive modelling applications, the nature of the output y that needs to be predicted or analysed could influence the architecture of the learning algorithm.

Explaining the details of how different ML techniques work is outside the scope of this paper. However, to provide an insight into how such algorithms function in layman's terms and the differences between them, we briefly present (hypothetical) supervised, unsupervised and reinforcement learning problems in the context of sports. These examples will assist the professionals but also applied researchers who work in sport to better understand the way that data scientists think so to facilitate talking to them about their approach and methodology, without requiring to dive deep into the details of the underlying analytics.

Supervised Learning: Predicting Player Injury

Many sports injuries (e.g., muscle strain) can be effectively treated or prevented if one is able to detect them early or predict the likelihood of sustaining them. There could be many different (combinations of) reasons/actions leading to injuries like muscle strain. For example, in the Australian Football League, some of hypotheses put forward leading to muscle strain include: muscle weakness and lack of flexibility, fatigue, inadequate warm-up, and poor lumbar posture ( Brockett et al., 2004 ). Detecting the patterns that can lead to such injuries is extremely important both for the safety of the players, and for the success and competitiveness of the team.

In a supervised learning scenario, data about the players would be collected from previous seasons including details such as the number of overall matches and consecutive matches they played, total time played in each match, categorised by age, number of metres run, whether or not they warmed up before the match, how many times they were tackled by other players, and so on , but more importantly, whether or not the players ended up injured and missed their next match.

The last point is very important as it is the principal difference between supervised learning and other approaches: the outcome (whether or not the player was injured) is known in the historical data that was collected from previous seasons. This historical data is then fed (with the outcome) to a machine learning algorithm with the objective of learning the patterns (combination of factors) which led to an injury (and usually assigning a probability of the likelihood of an injury given these patterns). Once these patterns are learnt, the algorithm or model is then tested on new (unseen data) to see if it performs well and indeed predicts/explains injury at a high level of accuracy (e.g., 70% of the time). If the accuracy of the model is not as required, the model is tuned (or trained with slightly different parameters) until it reaches the desired or acceptable accuracy. Note here that we did not single out a specific algorithm or technique to achieve the above. Indeed, this approach can be applied using many different ML algorithms such as Neural Networks, Decision Trees and regression models.

Unsupervised Learning: Fan Segmentation

We will use a sport business example to introduce the unsupervised learning approach. Most sports organisations keep track of historical data about their patrons who attended their sporting events, recording characteristics such as their gender, postcode, age, nationality, education, income, marital status, etc. A natural question of interest here is to understand if the different segments of customers/patrons will purchase different categories (e.g., price, duration, class etc.) of tickets.

Some AI algorithms are designed to help split the available data, so that each data point (historical ticket sale) sits in a group/class that is similar to the other data points (other sales) in that same class given the recorded features. The algorithm will then use some sort of a similarity or distance metric to classify the patrons according to the category of tickets that they might purchase.

This is different from how supervised learning algorithms, like those discussed in the previous section, work. As we described before, in supervised learning we instruct the algorithm with the outcome in advance while training it (i.e., we classify/label each observation based on the outcome: injury or no injury, cheap or expensive seats, …). In the unsupervised learning approach, there is no such labelling or classification of existing historical data. It is the mission of the unsupervised learning algorithm to discover (previously unnoticed) patterns in the input data and group it into (two or more) classes.

Imagine the following use case where an Australian Football League club aims to identify a highly profitable customer segment within its entire set of stadium attendees, with the aim to enhance its marketing operations. Mathematical models can be used to discover (segments of) similar customers based on variations in some customer attributes within and across each segment. A popular unsupervised learning algorithm to achieve such goal is the K-means clustering algorithm which finds the class labels from the data. This is done by iteratively assigning the data points (e.g., customers) from the input into a group/class based on the characteristics of this input. The essence is that the groups or classes to which the data points are assigned to are not defined prior to exploring the input data (although the number of groups or segments can be pre-defined) but are rather dynamically formed as the K-means algorithm iterates over the data points. In the context of customer segmentation, when presenting the mathematical model (K-means algorithm) with customer data, there is no requirement to label a portion (or any of) of this data into groups in advance in order to train the model as usually done in supervised models.

Reinforcement Learning: Simulations and Fantasy Sports

As mentioned before, in reinforcement learning, an algorithm (such as Q-learning and SARSA algorithms) learns how to complete a series of tasks (i.e., solve a problem) by interacting with an (artificial) environment that was designed to simulate the real environment/problem at hand. Unlike the case with supervised learning, the algorithm is not explicitly instructed about the right/accurate action in different states/conditions of the environment (or steps of problem it is trying to solve). But rather it incrementally learns such a protocol through reward maximisation.

In simple terms, reinforcement learning approaches represent problems using what are referred to as: an agent (a software algorithm), and a table of states and actions . When the agent executes an action, it transitions from one state to another and it receives a reward or a penalty (a positive or negative numerical score respectively) as a result. The reward/penalty associated with the action-state combination is then stored in the agent's table for future reference and refinement. The agent's goal is to take the action that maximises its reward. When the agent is still unaware of the expected rewards from executing a given action when at a given state, it takes a random action and updates its table following that action. After many (thousands of) iterations over the problem space, the agent's table holds (a weighted sum of) the expected values of the rewards of all future actions starting from the initial state.

Reinforcement learning has been applied to improve the selection of team formations in fantasy sports ( Matthews et al., 2012 ). Likewise, the use of reinforcement learning is prominent in online AI bots and simulators like chess, checkers, Go, poker, StarCraft, etc.

Finally, it is important to also note the existence of genetic or evolutionary algorithms, sometimes referred to as nature/bio-inspired algorithms. While such algorithms are not typically considered to be ML algorithms (but rather search techniques and heuristics), they are very popular in solving similar types of problems tackled by ML algorithms. In short, the idea behind such algorithms is to run (parallel) search, selection and mutation techniques, by going over possible candidate solutions of a problem. The solutions are gradually optimised until reaching a local (sub-optimal) or global maximum (optimal solution). To provide a high-level understanding of evolutionary algorithms, consider the following sequence of steps:

• We start by creating (a population of) initial candidate or random strategies/solutions to the problem at hand.

• We assess these candidate solutions (using a fitness function) and assign scores to each according to how well they solve the problem at hand.

• We then pick a selection of these candidate solutions that performed best at stage two above. We then combine ( crossbreed ) these together to generate ( breed) new solutions (e.g., take some attributes from one candidate solution and others from another candidate solution in order to come up with a new solution).

• We then apply random changes ( mutations ) to the resulting solutions from the previous step.

• We repeat the solution combination/crossbreeding process until a satisfactory solution is reached.

Evolutionary algorithms can be used as alternative means for training machine learning algorithms such as reinforcement learning algorithms and deep neural networks.

The Future of AI in Sport

There is no doubt that AI will continue to transform sports, and the ways in which we play, watch and analyse sports will be innovative and unexpected. In fact, machine learning has drastically changed the way we think about match strategies, player performance analytics but also how we track, identify and learn about sport consumers. A Pandora's box of ethical issues is emerging and will increasingly need to be considered when machines invade the traditionally human centred and naturally talented athlete base of sport. It is unlikely that AI will completely replace coaches and human experts, but there is no doubt that leveraging the power of AI will provide coaches and players with a big advantage and lead over those who only rely on human expertise. It will also provide sport business managers with deeper, real time insights into the behaviours, needs and wants of sport consumers and in turn AI will become a main producer of sport content that is personalised and custom made for individual consumers. But human direction and intervention seems to be, at least in the near future, still essential working towards elite sport performance and strategic decision making in sport business. The sporting performance on the field is often produced as an entertainment spectacle, where the sporting context is the platform for generating the business of sport. Replacing referees with automated AI is clearly possible and increasingly adopted in various sports, because it is more accurate and efficient, but is it what the fans want?

What might the future of sport with increasingly integrated AI look like? Currently, most of the research in AI and sports is specialised. That is to provide performance or business solutions and solve specific on and off field problems. For instance, scientists have successfully devised solutions to tackle problems like player performance measurement, and quantifying the effect of a player/team on demand for gate attendance. Nevertheless, our research has not identified studies (yet) that provide a 360-degree analysis on, for example, the absolute value of an athlete by taking into account all the dimensions of his or her performance on how much business can be developed, for example in regard to ticket sales or endorsement deals.

One of the main challenges to achieve such a comprehensive analysis is mainly due to the fact that data about players and teams, and commercial data such as ticket sales and attendance numbers, are kept proprietary and are not made public to avoid providing other parties with competitive information. Moreover, privacy is an important consideration as well. Regulations about data privacy and leakage of personal identification details must be put in place to govern the use and sharing of sports (performance and consumption) data. Data ownership, protection, security, privacy and access will all drive the need for comprehensive and tight legislation and regulation that will strongly influence the speed and comprehensiveness of the adoption of AI in sport. To that end, it is worth considering privacy and confidentiality implications independently when studying the leagues' journey of AI adoption compared to that of individual teams and ultimately the individual players. Eventually, the successful adoption of AI in a sports league will likely depend on the teams in that league and their players to be willing to share proprietary data or insights with other teams in the league. Performance data of players in particular is becoming a hot topic of disputation. It may well be AI that will determine the bargaining power of players and their agents in regard to the value of their contracts. As an extension of this it will then also be AI providing the information that will determine if players are achieving the performance objectives set by coaches and as agreed to in contracts. In other words, confidentiality and ownership of league, team or player level data will become an increasing bone of legal contention and this will be reflected in the complexity of contractual agreements and possible disputes in the change rooms and on the field of play. Being in control of which data can or cannot, and will or will not, be used is at stake.

From an economic perspective, relying on artificial algorithms could increase the revenue of sports organisations and event organisers when enabled to apply efficient variable and dynamic pricing strategies and build comprehensive and deep knowledge consumer platforms. Different types of ML algorithms can be adopted to deliver more effective customer marketing via personalisation and to increase sales funnel conversion rates.

Finally, for a window on the future of data privacy, it might be useful to return to baseball where the addiction to big data started its spread across the high-performance sport industry. Hattery (2017 , p. 282) explains that in baseball “using advanced data collection systems … the MLB teams compete to create the most precise injury prediction models possible in order to protect and optimise the use of their player-assets. While this technology has the potential to offer tremendous value to both team and player, it comes with a potential conflict of interest. Players' goals are not always congruent with those of the organisation: the player strives to protect his own career while the team is attempting to capitalise on the value of an asset. For this reason, the player has an interest in accessing data that analyses his potential injury risk. This highlights a greater problem in big data: what rights will individuals possess regarding their own data points?”

This privacy issue can be further extended to the sport business space Dezfouli et al. (2020) have shown how AI can be designed to manipulate human behaviour. Algorithms learned from humans' responses who were participating in controlled experiments. The algorithms identified and targeted vulnerabilities in human decision-making. The AI succeeded in steering participants towards executing particular actions. So, will AI one day be shaping the spending behaviour of sports fans by exploiting their fan infused emotional vulnerabilities and monitoring their (for example) gambling inclinations? Will AI sacrifice the health of some athletes in favour of the bigger team winning the premiership? Or is this already happening? Time will tell.

Data Availability Statement

The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author.

Author Contributions

NC and HW had major contribution to the writing of this manuscript. NC contributed to the writing of the parts around artificial intelligence and machine learning and provided examples of these. HW shaped the scope of the manuscript and wrote and edited many of its sections particularly the introduction and the discussion. Both authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

1. ^ For conferences and published articles on AI and sports analytics see Swartz (2020) .

2. ^ Note that such function is also found in regression techniques where the weights/coefficients are unknown. In ML, it is usually the case where both the function and its weights are unknown and are determined using various search techniques and algorithms.

Anderson, J. A. (1995). An introduction to Neural Networks . Cambridge, MA: MIT Press.

Google Scholar

Atiković, A., Kamenjašević, E., Nožinović, M. A., Užičanin, E., Tabaković, M., and Curić, M. (2020). Differences between all-around results in women's artistic gymnastics and ways of minimizing them. Balt. J. Health Phys. Act. 12, 80–91. doi: 10.29359/BJHPA.12.3.08

CrossRef Full Text | Google Scholar

Bartlett, R. (2006). Artificial intelligence in sports biomechanics: new dawn or false hope? J. Sports Sci. Med. 5, 474–479.

PubMed Abstract | Google Scholar

Beal, R., Norman, T. J., and Ramchurn, S. D. (2019). Artificial intelligence for team sports: a survey. Knowl. Eng. Rev. 34. doi: 10.1017/S0269888919000225

PubMed Abstract | CrossRef Full Text | Google Scholar

Berri, D. J., Schmidt, M. B., and Brook, S. L. (2004). Stars at the gate: the impact of star power on NBA gate revenues. J. Sports Econom. 5, 33–50. doi: 10.1177/1527002503254051

Bonaccorso, G. (2017). Machine Learning Algorithms . Birmingham: Packt Publishing Ltd.

Brandes, L., Franck, E., and Nuesch, S. (2008). Local heroes and superstars: an empirical analysis of star attraction in German soccer. J. Sports Econom. 9, 266–286. doi: 10.1177/1527002507302026

Brockett, C. L., Morgan, D. L., and Proske, U. W. E. (2004). Predicting hamstring strain injury in elite athletes. Med. Sci. Sports Exerc. 36, 379–387. doi: 10.1249/01.MSS.0000117165.75832.05

Brown, N., and Sandholm, T. (2019). Superhuman AI for multiplayer poker. Science 365, 885–890. doi: 10.1126/science.aay2400

Campbell, M., Hoane Jr, A. J., and Hsu, F. H. (2002). Deep blue. Artif. Intell. 134, 57–83. doi: 10.1016/S0004-3702(01)00129-1

Chmait, N. (2017). Understanding and measuring collective intelligence across different cognitive systems: an information-theoretic approach. in IJCAI (Melbourne), 5171–5172.

Chmait, N., Dowe, D. L., Li, Y. F., Green, D. G., and Insa-Cabrera, J. (2016). Factors of collective intelligence: how smart are agent collectives? in Proceedings of the Twenty-second European Conference on Artificial Intelligence (Prague), 542–550.

Chmait, N., Robertson, S., Westerbeek, H., Eime, R., Sellitto, C., and Reid, M. (2020a). Tennis superstars: the relationship between star status and demand for tickets. Sport Manag. Rev . 23, 330–347. doi: 10.1016/j.smr.2019.03.006

Chmait, N., Westerbeek, H., Eime, R., Robertson, S., Sellitto, C., and Reid, M. (2020b). Tennis influencers: the player effect on social media engagement and demand for tournament attendance. Telemat Inform. 50:101381. doi: 10.1016/j.tele.2020.101381

Coates, D., and Humphreys, B. R. (2012). Game attendance and outcome uncertainty in the National Hockey League. J. Sports Econom. 13, 364–377. doi: 10.1177/1527002512450260

Dezfouli, A., Nock, R., and Dayan, P. (2020). Adversarial vulnerabilities of human decision-making. Proc. Nat. Acad. Sci. U.S.A. 117, 29221–29228. doi: 10.1073/pnas.2016921117

Fister Jr, I., Ljubič, K., Suganthan, P. N., Perc, M., and Fister, I. (2015). Computational intelligence in sports: challenges and opportunities within a new research domain. Appl. Math. Comput. 262, 178–186. doi: 10.1016/j.amc.2015.04.004

Ghosh, J. K., Delampady, M., and Samanta, T. (2007). An Introduction to Bayesian Analysis: Theory and Methods . Berlin: Springer Science and Business Media.

Hakes, J. K., and Sauer, R. D. (2006). An economic evaluation of the Moneyball hypothesis. J. Econ. Perspect. 20, 173–186. doi: 10.1257/jep.20.3.173

Hattery, M. (2017). Major League Baseball players, big data, and the right to know: the duty of Major League Baseball teams to disclose health modelling analysis to their players. Marquette Sports Law Rev. 28, 257–283. https://scholarship.law.marquette.edu/sportslaw/vol28/iss1/9/

Hernández-Orallo, J., Baroni, M., Bieger, J., Chmait, N., Dowe, D. L., Hofmann, K., et al. (2017). A new AI evaluation cosmos: ready to play the game? AI Mag. 38, 66–69. doi: 10.1609/aimag.v38i3.2748

Jane, W.-J. (2016). The effect of star quality on attendance demand: the case of the National Basketball Association. J. Sports Econom. 17, 396–417. doi: 10.1177/1527002514530405

Jewell, R. T. (2017). The effect of marquee players on sports demand: the case of US Major League Soccer. J. Sports Econom. 18, 239–252. doi: 10.1177/1527002514567922

Konjer, M., Meier, H. E., and Wedeking, K. (2017). Consumer demand for telecasts of tennis matches in Germany. J. Sports Econom. 18, 351–375. doi: 10.1177/1527002515577882

Krause, L. (2019). Exploring the influence of practice design on the development of tennis players (Doctoral dissertation). Victoria University, Footscray, VIC, Australia.

Lapham, A. C., and Bartlett, R. M. (1995). The use of artificial intelligence in the analysis of sports performance: a review of applications in human gait analysis and future directions for sports biomechanics. J. Sports Sci. 13, 229–237. doi: 10.1080/02640419508732232

Lenten, L. J. (2012). Comparing attendances and memberships in the Australian Football League: the case of hawthorn. Econ Labour Relat. Rev. 23, 23–38. doi: 10.1177/103530461202300203

Lewis, M. (2004). Moneyball: The Art of Winning an Unfair Game . New York, NY: WW Norton and Company.

Lewis, M., and Yoon, Y. (2016). An empirical examination of the development and impact of star power in Major League Baseball. J. Sports Econom. 19, 155–187. doi: 10.1177/1527002515626220

Matthews, T., Ramchurn, S., and Chalkiadakis, G. (2012). Competing with humans at fantasy football: Team formation in large partially-observable domains. in Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 26 (Vancouver, BC), 1394–1400.

McCabe, A., and Trevathan, J. (2008). Artificial intelligence in sports prediction. in Fifth International Conference on Information Technology: New Generations (IEEE: Las Vegas, NV), 1194–1197. doi: 10.1109/ITNG.2008.203

Nadikattu, R. R. (2020). Implementation of new ways of artificial intelligence in sports. J. Xidian Univ. 14, 5983–5997. doi: 10.2139/ssrn.3620017

Novatchkov, H., and Baca, A. (2013). Artificial intelligence in sports on the example of weight training. J. Sports Sci. Med. 12, 27–37.

Ormiston, R. (2014). Attendance effects of star pitchers in major league baseball. J. Sports Econom. 15, 338–364. doi: 10.1177/1527002512461155

Paton, D., and Cooke, A. (2005). Attendance at county cricket: an economic analysis. J. Sports Econom. 6, 24–45. doi: 10.1177/1527002503261487

Piccinotti, D. (2021). Open Loop Planning for Formula 1 Race Strategy Identification. Menlo Park, CA: Association for the Advancement of Artificial Intelligence.

Ratiu, O. G., Badau, D., Carstea, C. G., Badau, A., and Paraschiv, F. (2010). Artificial intelligence (AI) in sports, in Proceedings of the 9th WSEAS International Conference on Artificial Intelligence, Knowledge Engineering, and Data Bases (Cambridge, UK), 93–97.

Risi, S., and Preuss, M. (2020). From chess and Atari to StarCraft and beyond: how game AI is driving the world of AI. KI-Künstliche Intell. 34, 7–17. doi: 10.1007/s13218-020-00647-w

Schaeffer, J., Burch, N., Björnsson, Y., Kishimoto, A., Müller, M., Lake, R., et al. (2007). Checkers is solved. science 317, 1518–1522. doi: 10.1126/science.1144079

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., et al. (2017). Mastering the game of go without human knowledge. Nature 550, 354–359. doi: 10.1038/nature24270

Swartz, T. B. (2020). Where should I publish my sports paper? Am. Stat. 74, 103–108. doi: 10.1080/00031305.2018.1459842

Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., et al. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354. doi: 10.1038/s41586-019-1724-z

Whitaker, G. A., Silva, R., Edwards, D., and Kosmidis, I. (2021). A Bayesian approach for determining player abilities in football. J. R. Stat. Soc. Series C 70, 174–201. doi: 10.1111/rssc.12454

Yang, T. Y., and Swartz, T. (2004). A two-stage Bayesian model for predicting winners in major league baseball. J. Data Sci. 2, 61–73. doi: 10.6339/JDS.2004.02(1).142

Keywords: artificial intelligence, machine learning, sports business, sports analytics, sport research, future of sports

Citation: Chmait N and Westerbeek H (2021) Artificial Intelligence and Machine Learning in Sport Research: An Introduction for Non-data Scientists. Front. Sports Act. Living 3:682287. doi: 10.3389/fspor.2021.682287

Received: 18 March 2021; Accepted: 15 November 2021; Published: 08 December 2021.

Reviewed by:

Copyright © 2021 Chmait and Westerbeek. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Nader Chmait, nader.chmait@vu.edu.au

This article is part of the Research Topic

The Future of Sport Business

• Harvard Business School →
• Faculty & Research →

Publications

• Global Research Centers
• Case Development
• Initiatives & Projects
• Research Services
• Seminars & Conferences
• Publications →

Show Results For

• All HBS Web  (115,710)
• Faculty Publications  (58,209)

AIandMachineLearning →

No results found in faculty publications.

• Were any results found in one of the other content buckets on the left?
• Try removing some search filters.
• Use different search filters.

Artificial intelligence, machine learning and deep learning

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Artificial intelligence and machine learning

• Fundamentals
• Open access
• Published: 09 November 2022
• Volume 32 , pages 2235–2244, ( 2022 )

Cite this article

You have full access to this open access article

• Niklas Kühl   ORCID: orcid.org/0000-0001-6750-0876 1 ,
• Max Schemmer 1 ,
• Marc Goutier 2 &
• Gerhard Satzger 1  

28k Accesses

27 Citations

3 Altmetric

Explore all metrics

Within the last decade, the application of “artificial intelligence” and “machine learning” has become popular across multiple disciplines, especially in information systems. The two terms are still used inconsistently in academia and industry—sometimes as synonyms, sometimes with different meanings. With this work, we try to clarify the relationship between these concepts. We review the relevant literature and develop a conceptual framework to specify the role of machine learning in building (artificial) intelligent agents. Additionally, we propose a consistent typology for AI-based information systems. We contribute to a deeper understanding of the nature of both concepts and to more terminological clarity and guidance—as a starting point for interdisciplinary discussions and future research.

Avoid common mistakes on your manuscript.

Introduction

Artificial Intelligence (AI) has been named as one of the most recent, fundamental developments of the convergence in electronic markets (Alt, 2021 ) and has become an increasingly relevant topic for information systems (IS) research (Abdel-Karim et al., 2021 ; Alt, 2018 ). While a large body of literature is concerned with designing AI to mimic and replace humans (Dunin-Barkowski, 2020 ; Fukuda et al., 2001 ), IS research in general, and decision support systems (DSS) research in particular, emphasize the support of humans with AI (Arnott & Pervan, 2005 ). Recent research in hybrid intelligence (HI) and human-AI collaboration offers a promising path in synthesizing AI research across different fields (Dellermann, 2019 ): The ultimate goal of HI is to leverage the individual advantages of both human and artificial intelligence to enable synergy effects (James & Paul, 2018 ) and to achieve complementarity (Hemmer et al., 2021 ).

However, in many cases in both research and practice, AI is simply equated with the concept of machine learning (ML)—negatively impacting terminological precision and effective communication. Ågerfalk ( 2020 , p.2) emphasizes that differentiating between AI and ML is especially important for IS research: “Is it not our responsibility as IS scholars to bring clarity to the discourse rather than contributing to its decline? (…) It would mean to distinguish between different types of AI and not talk of AI as synonymous with ML, which in itself is far from a monolithic concept.”

The practical relevance of a clear understanding is underlined by observing confusion and misuse of the terms AI and ML: During Mark Zuckerberg’s U.S. senate hearing in April 2018, he stressed that Facebook had “AI tools to identify hate speech” as well as “terrorist propaganda” (The Washington Post, 2018 ). Researchers, however, would usually describe tasks identifying specific social media platform instances as classification tasks in the field of (supervised) ML (Waseem & Hovy, 2016 ). The increasing popularity of AI (Fujii & Managi, 2018 ) has led to the term often being used interchangeably with ML. This does not only hold true for the statement of Facebook’s CEO above, but also across various theoretical and application-oriented contributions in recent literature (Brink, 2017 ; ICO, 2017 ; Nawrocki et al., 2018 ). Camerer ( 2017 ) even mentions that he still uses AI as a synonym for ML despite knowing it is inaccurate.

As the remainder of this paper shows, both concepts are not identical—although in many cases both terms will appear in the same context. Such ambiguity might lead to multiple imprecisions in both research and practice when conversing about the relevant concepts, methods, and results. This is especially important in IS research—being interdisciplinary by nature (D’Atri et al., 2008 ). Ultimately, misuse can either lead to fundamental misunderstandings (Carnap, 1955 ) or to research that ought to be undertaken not being conducted (Davey & Cope, 2008 ; Lange, 2008 ). After all, misunderstandings can potentially lead to low perceived trustworthiness of AI (Thiebes et al., 2021 ).

It seems surprising that despite the frequent use of the terms, there is hardly any helpful academic delineation—apart from the notion that ML is a (not well-defined) subset of AI (Campesato, 2020 ), comparable to other possible subdisciplines of AI: Expert systems, robotics, natural language processing, machine vision, and speech recognition (Collins et al., 2021 ; Dejoux & Léon, 2018 ). Consequently, this paper aims to shed light on the relationship between the two concepts: We analyze the role of ML in AI and, more precisely, in intelligent agents, which are defined by their capability to sense and act in an environment (Schleiffer, 2005 ). We do so by taking an ML perspective on intelligent agents’ capabilities and their relevant implementation—with IS research in mind. To this end, we review the relevant literature for both terms and synthesize and conceptualize the results.

Our article’s contributions are twofold: First, we identify different contributions of ML to intelligent agents as specific AI instantiations. We base this on an expansion of the existing AI framework by Russell and Norvig ( 2020 ) — explicitly breaking down intelligent agents’ capabilities into separate “execution” and “learning” capabilities. Second, we develop a typology to provide a common terminology for AI-based information systems, where we conceptualize which systems employ ML—and which do not. The result should provide guidance when designing and analyzing systems.

Next, in Section “ Terminology ”, we review relevant literature in the fields of AI and ML. In Section “ The role of rational agents in information systems ”, we then analyze the capabilities of intelligent agents in more depth and examine the role of ML in them. Section “ Towards a typology for machine learning in AI systems ” develops a framework and typology to differentiate the terms AI and ML and to explain their relationship. In Section “ Conclusion ”, we conclude with a summary.

• Terminology

Over the last decade, both terms, artificial intelligence (AI) and machine learning (ML), have enjoyed increasing popularity in information systems (IS) research. An analysis of the “AIS Senior Scholars’ Basket” Footnote 1 journals since 2000, Footnote 2 illustrates how the occurrences of both terms increased in titles, abstracts, and keywords (Fig.  1 ). While over the last 21 years, we observe a small but constant number of publications covering AI-related topics, ML only gained relevance in the literature after 2017: The late reflection of ML—despite of the earlier adoption and spread in industry (Brynjolfsson & Mcafee, 2017 )—may raise questions about whether IS has picked up the topic early enough.

Appearance of the terms “artificial intelligence” and “machine learning” and in AIS Senior Scholars’ Basket journals

As the analysis demonstrates, the two terms do exist for quite some time, while their related subjects are highly and increasingly topical now. In this section, we will elaborate on the meaning of the terms.

• Artificial intelligence

In 1956, a Dartmouth workshop, led by Minsky and McCarthy, coined the term “artificial intelligence” (McCarthy et al., 1956 ) —later taking in contributions from a variety of different research disciplines, such as computer science (K. He et al., 2016 ) and programming (Newell & Simon, 1961 ), neuroscience (Ullman, 2019 ), robotics (Brady, 1984 ), linguistics (Clark et al., 2010 ), philosophy (Witten et al., 2011 ), and futurology (Koza et al., 1996 ). While the terminology is not well defined across disciplines, even within the IS domain definitions do vary widely; Collins et al. ( 2021 ) provide a comprehensive overview. Recent AI definitions transfer the human intelligence concept to machines in its entirety as “the ability of a machine to perform cognitive functions that we associate with human minds, such as perceiving, reasoning, learning, interacting with the environment, problem solving, decision-making, and even demonstrating creativity” (Rai et al., 2019 , p.1). Still, over the last decades various debates have been raging on the depth and objectives of AI. These two dimensions span the space for different AI research streams in computer science and IS that were categorized by Russell and Norvig ( 2020 ): On the one hand (depth dimension), it may target either the thought process or a concrete action ( thinking vs. acting ); on the other hand (objective dimension), it may try to either replicate human decision making or to provide an ideal, “most rational” decision ( human-like vs. rational decision ). The resulting research streams are depicted in Table  1 .

According to the cognitive modeling (i.e., thinking humanly) stream, AI instantiations must be “machines with a mind” (Haugeland, 1989 ) that perform human thinking (Bellman, 1978 ). Not only should they arrive at the same output as a human when given the same input, but also apply the same reasoning steps leading to this conclusion (Newell & Simon, 1961 ). The laws of thought stream (i.e., thinking rationally) requires AI instantiations to arrive at a rational decision despite what a human might come up with. AI must therefore adhere to the laws of thought by using logic-based computational models (McDermott & Charniak, 1985 ). The Turing test stream (i.e., acting humanly) implies that AI must act intelligently when interacting with humans. To accomplish such tasks, AI instantiations must perform human tasks at least as well as humans (Rich & Knight, 1991 ), which can be tested via the Turing test (Turing, 1950 ). Finally, the rational agent stream considers AI as a rational (Russell & Norvig, 2020 ) or intelligent (Poole et al., 1998 ) agent. Footnote 3 This agent does not only act autonomously, but also with the objective of achieving the rationally ideal outcome.

• Machine learning

Many researchers perceive ML as an (exclusive) part of AI (Collins et al., 2021 ; Copeland, 2016 ; Ongsulee, 2017 ). In general, learning is a key facet of human cognition (Neisser, 1967 ). Humans process a vast amount of information by utilizing abstract knowledge that helps them to better understand incoming input. Owing to their adaptive nature, ML models can mimic a human being’s cognitive abilities (Janiesch et al., 2021 ): ML describes a set of methods commonly used to solve a variety of real-world problems with the help of computer systems, which can learn to solve a problem instead of being explicitly programmed to do so (Koza et al., 1996 ). For instance, instead of explicitly telling a computer system which words within an tweet would indicate it to contain a customer need, the system (given a sufficient set of training samples) learns the typical patterns of words and their combination which results in a need classification (Kühl et al., 2020 ).

In general, we differentiate between unsupervised , supervised , and reinforcement ML. Unsupervised ML comprises methods that reveal previously unknown patterns in data. Consequently, unsupervised learning tasks do not necessarily have a “correct” solution, as there is no ground truth (Wang et al., 2009 ).

Supervised ML refers to methods that allow the building of knowledge about a given task from a series of examples representing “past experience” (Mitchell, 1997 ). In the learning process, no manual adjustment or programming of rules or strategies to solve a problem is required, i.e., the model is capable to learn “by itself”. In more detail, supervised ML methods always aim to build a model by applying an algorithm to a set of known data points to gain insight into an unknown set of data (Hastie et al., 2017 ): Known data points are semantically labeled to create a target for the ML model. So-called semi-supervised learning combines elements from supervised and unsupervised ML by jointly using labeled and unlabeled data (Zhu, 2005 ).

Reinforcement learning refers to methods that are concerned with teaching intelligent agents to take those kinds of actions that increase their cumulative reward (Kaelbling et al., 1996 ). It differs from supervised learning in that no correctly matched features and targets are required for training. Instead, rewards and penalties allow the model to continuously learn over time. The focus is on a trade-off between the exploration of the uncharted environment and the exploitation of the existing knowledge base.

The role of rational agents in information systems

To further elaborate on the role of ML within AI, we need to take a clear perspective on the different definitions of AI to be beneficial to IS research. IS traditionally utilizes ML in predictive analytics tasks within (intelligent) decision support systems (DSS) (Arnott & Pervan, 2005 ; Müller et al., 2016 ) where the goal is to generate the best possible outcome (Arnott, 2006 ; Hunke et al., 2022 ; Power et al., 2019 ). As Phillips-Wren et al. ( 2019 , p.63) emphasize, DSS “should help the decision-maker think rationally”. The perspective of rationality is also endorsed by other researchers in the field (Bakos & Treacy, 1986 ; Dellermann, 2019 ; Kloör et al., 2018 ; Power et al., 2019 ; Schuetz & Venkatesh, 2020 ). Thus, in the following we will explore the relationship between ML and AI in IS from the lens of the rational agent stream as discussed above. Furthermore, we will focus on supervised ML as it is the most common type of ML (Jordan & Mitchell, 2015 ). In the remainder of this section, we will first distinguish different types of (rational) agents and then use the insights to differentiate between the necessary layers when designing them as part of information systems.

Types of rational agents

According to the selected research stream, intelligence manifests itself in how rational agents act. Five features characterize agents in general: they “operate autonomously, perceive their environment, persist over a prolonged time period, adapt to change, and create and pursue goals” (Russell & Norvig, 2020 , p.4). An agent defines its action, not for itself, but within the environment it operates and interacts with. It recognizes the environment through its sensors, relies on an agent program to handle and digest input data, and performs actions via actuators. A rational agent targets to achieve the highest expected outcome according to one or multiple objective performance measures—which are based on current and past knowledge of the environment and possible actions. For example, a rational agent within a medical diagnosis system aims to maximize the health of a patient measured via blood pressure, heart rate, and blood oxygen (potentially while minimizing the financial costs of a treatment as a secondary condition) (Grosu, 2022 ).

The agent’s conceptualization and surroundings are summarized in the agent-environment framework. It consists of three components: an agent, an environment, and a goal. Intelligence is the measurement of the “agent’s ability to achieve goals in a wide range of environments” (Legg & Hutter, 2007 , p. 12). The agent obtains input through perceptions that the environment generates. Observations of the environment are one type of perception, while others are reward signals that indicate how well the agents’ goals have been achieved. Based on these input signals, the agent decides to perform actions, which are subsequently communicated back as signals to the environment.

Rational agents in information system architectures

As we investigate the role of ML in AI for IS research, we also need — apart from the theoretical and definitory aspects of agents — to consider how the functionality of a rational agent is reflected in an IS architecture. The implementation of agents is a key step to embed their functionality into practical, real-world (intelligent) information systems in general or into DSS specifically (Gao & Xu, 2009 ; Zhai et al., 2020 ). Any rational agent needs to be capable of at two least two tasks: cognition (Lieto et al., 2018 ) and (inter)action with the environment (Russell & Norvig, 2020 ). If we map these capabilities to system design terms, then acting capabilities are the ones built into a frontend , while the cognitive capabilities are embedded in a backend .

The frontend as the interface to the environment may take various forms; it may be designed as a very abstract, machine-readable web interface (Kühl et al., 2020 ), a human-readable application (Engel et al., 2022 ; Hirt et al., 2019 ), or even a humanoid template with elaborated expression capabilities (Guizzo, 2014 ). For the frontend to interact with the environment, two technical components are required: sensors and actuators. Sensors detect events or changes in the environment and forward the information via the frontend to the backend. They can, for instance, read the signals within an industrial process network (Hein et al., 2019 ), read visuals of an interaction with a human (Geller, 2014 ), but also perceive a keystroke input (Russell & Norvig, 2020 ). Actuators , on the other hand, are components responsible for moving, controlling, or displaying content. While sensors merely process information, actuators act, for instance, by automatically making bookings (Neuhofer et al., 2021 ) or changing a humanoid’s facial expressions (Berns & Hirth, 2006 ). One could argue that the Turing test (Turing, 1950 ) takes place at the environment’s interaction with the frontend, or, more precisely, when sensors and actuators are combined in a way to test the agent’s AI for acting humanly .

The backend provides the required functionalities to depict an intelligent agent’s cognitive capabilities. More precisely, this executing backend allows the agent to draw on its built-in knowledge. The backend translates signals from the frontend and transforms them into signals sent back to the frontend as a response by executing actions. In some cases, there is an additional component modifying this response function over time, and thus modifying the execution part of the backend. We call this the learning part of the backend as depicted in Fig.  2 . Within the next subsections, we will further elaborate this framework and its components.

Conceptual framework describing the general architecture for intelligent agents in AI-based information systems

The role of machine learning in rational agents

In terms of supervised ML, we need to further differentiate between the process task of building (training) adequate ML models (Witten et al., 2011 ) and the one of executing the deployed models (Chapman et al., 2000 ). To further understand ML’s role in intelligent agents, we partition the agent’s cognition layer into a learning sublayer (model building) and an executing sublayer (model execution). Footnote 4 We, therefore, regard the implementation required by the learning sublayer as the learning backend , while the executing backend denotes the executing sublayer.

The learning backend first dictates if the intelligent agent is able to learn, and, second, how it does so — with respect to the algorithms it actually uses, the type of data processing it applies, and the handling of concept drift (Gama et al., 2014 ). Using the terminology of Russell and Norvig ( 2020 ), we distinguish two different types of intelligent agents: simple-reflex agents and learning agents . This differentiation holds explicitly in terms of a ML perspective on AI because it considers whether the underlying models in the cognition layer are trained just once and after that never touched (simple-reflex), or whether they are continuously updated to be adaptive (learning). Related work provides suitable examples of both. Kitts and Leblanc ( 2004 ) build a bidding agent for digital auctions as a simple-reflex agent: While building and testing the model for the agent may show convincing results, the system’s adaptive learning after deployment could be critical. Other examples of agents with models trained just once are common in different areas, for example, in terms pneumonia warning for hospitals (Oroszi & Ruhland, 2010 ), the (re)identification of pedestrians (Z. Zheng et al., 2017 ), and object annotation (Jorge et al., 2014 ). On the other hand, recent literature also provides examples of learning agents. Mitchell et al. ( 2015 ) present the concept of “never-ending learning” agents that strongly focus on continuously building and updating models in agents. Neuhofer et al. ( 2015 ) suggest an agent capable of personalization through a continuous learning processes of guest information for digital platforms, which an example of such an agent. Other examples include agents capable of making recommendations on music platforms (Liebman et al., 2015 ), regulating heat pump thermostats (Ruelens et al., 2015 ), acquiring collective knowledge across different tasks (Rostami et al., 2017 ), and learning the meanings of words (Yu et al., 2017 ). The choice of the learning type in agents (simple-reflex vs. learning agent) influences the agent’s general overall design and the contribution of ML.

As a result from the layers of agents and types of learning, our conceptual framework combining both is shown in Fig. 2 . Regarding the previously mentioned ML methods, supervised ML can be the basis for either simple-reflex or learning agents, depending on whether the learning backend exists and on its feedback to the agent’s knowledge base. In terms of reinforcement learning, the agent, by definition, is a learning agent. However, there are also examples of where an agent functions without the utilization of ML—because the execution is based on rules (H. Wang et al., 2005 ), formulas (Billings et al., 2002 ) or other methods (Abasolo & Gomez, 2000 ). From this perspective, this means there can be AI without ML.

Towards a typology for machine learning in AI systems

Based on the differentiation between simple-reflex and learning agents, we can now derive a typology for IS research. We refer to IS systems as static AI-based systems if they employ simple reflex agents that may be based on a model trained with ML. Adaptive AI-based systems , though, use learning agents, i.e., do have a learning backend— that may be based on ML, but alternatively also could be based, e.g., on rule-based knowledge representation. We, thus, propose the typology (as depicted in Table  2 ) for AI-based IS along the two dimensions: the existence of an ML-base for the executing backend and the existence of a learning backend.

We illustrate these findings in concrete IS research examples: Static AI systems are characterized by an executing backend which is based on algorithms not classified as ML and they lack a learning backend, i.e. they have a fixed response model (Chuang & Yadav, 1997 ). The executing backend of such systems is based on rules (like nested if-else statements), formulas (like mathematic equations describing a phenomena) or algorithms (like individual formal solution descriptions for specific problems). As an example for such systems, Hegazy et al. ( 2005 ) build a static AI system based on a self-developed algorithm and evaluate its performance within a cybersecurity context by simulating multiple attacks. Another example is provided by Ritchie ( 1990 ) who has developed an architecture and an instantiation of a static AI system for a traffic management platform.

In contrast, a static ML-based AI system has an executing backend which is based on ML. An example is provided in S. He et al. ( 2018 ). The authors develop an artifact to classify marketing on Twitter in either defensive or offensive marketing and show convincing prediction results. While their work did not aim at designing a productive artifact and is rather focused on showing the general feasibility of the approach, they choose a static ML-based AI system—which, however, might not be sufficient for permanent use: After the release of the article in 2018, Twitter changed its tweet size from 140 to 280 characters, thus changing the environment. It would be interesting to see how the developed model would need to adapt to this change. As another example, Samtani et al. ( 2017 ) build a model to identify harmful code snippets, typically utilized by hackers. They show how to design an artifact that can detect these code assets accurately for a proactive cyber threat intelligence. However, also in this case the environment and the assets of the hackers could and will change over time.

Adaptive AI systems , which are not based on ML, do comprise an executing backend with the flexibility to dynamically adapt the model to changing environments. This type of system is oftentimes enabled through the interaction between humans and AI systems. Most of the times, the system provides means and triggers for updates, while the human provides “manually encoded” knowledge updates. For example, Zhou et al. ( 2009 ) implement an adaptive AI system for pipeline leak detection which is based on a rule-based expert system and offers means to update the system online. In another example, Hatzilygeroudis and Prentzas ( 2004 ) develop an adaptive AI system to support the teaching process which has a specific component for knowledge updates. Both examples are inherently knowledge-based, but are explicitly designed to allow and force updates—although not on the basis of ML.

Finally, adaptive ML-based AI-systems implement learning in both sublayers of the cognition layer. For example, Q. Zheng et al. ( 2013 ) design a reinforcement-learning-based artifact to obtain information from hidden parts (“deep web”) of the internet. As their developed system perceives its current state and selects an action to submit to the environment (the deep web), the system continuously learns and builds up experience. In another example, Ghavamipoor and Hashemi Golpayegani ( 2020 ) build an adaptive ML-based AI system to predict the necessary service quality level and adapt an e-commerce system accordingly. As their system is continuously learning, their results show the total profits improve through effective cost reduction and revenue enhancement.

In this article, we clarify the relationship of machine learning (ML) in artificial intelligence (AI), particularly in intelligent agents, for the field of information systems research. Based on a rational agent view, we differentiate between AI agents capable of continuously improving as well as those who are static. Within these agents as instantiations of artificial intelligence, (supervised) ML can serve to support in different ways: either to contribute a once-trained model to define a static response pattern or to provide an adaptive model to realize dynamic behavior. As we point out, both could also be realized without the application of ML. Thus, “ML” and “AI” are not terms that should be used interchangeably—but as a conscious choice. Without question, ML is an important driver of AI, and the majority of modern AI cases will utilize ML. However, as we illustrate, there can be cases of AI without ML (e.g., based on rules or formulas).

This distinction enables our proposed framework to apply an intelligent agent’s perspective on AI-based information systems, enabling researchers to differentiate the existence and function of ML in them. Interestingly, as of today, many AI-based information systems remain static, i.e. employ once-trained ML models (Kühl et al., 2021 ). With increasing focus on deployment and life cycle management, we will see more adaptive AI-based systems that sense changes in the environment and use ML to learn continuously (Baier et al., 2019 ). Our framework and the resulting typology should allow IS researchers and practitioners to be more precise when referring to ML and AI, as it highlights the importance of not using the terms interchangeably but clarifying the role ML plays in AI’s system design.

As of March 2022, see https://aisnet.org/page/SeniorScholarBasket , last accessed 16.05.2022.

We start with the year 2000, as it was the last point in time when a journal (JAIS) was added to AIS Senior Scholars’ Basket.

In this case, the terms rational and intelligent are used interchangeably in related work (Gama et al., 2014 ; Koza et al., 1996 ; Russell & Norvig, 2020 ).

Russel and Norvig indicate a related relationship by differentiating between learning elements and performance elements (Russell & Norvig, 2020 ).

Abasolo, J. M., & Gomez, M. (2000). MELISA: An ontology-based agent for information retrieval in medicine. Proceedings of the 1st international workshop on the semantic web (SemWeb2000) , 73–82.

Abdel-Karim, B. M., Pfeuffer, N., & Hinz, O. (2021). Machine learning in information systems - a bibliographic review and open research issues. Electronic Markets, 31 (3), 643–670. https://doi.org/10.1007/s12525-021-00459-2

Article   Google Scholar  

Ågerfalk, P. J. (2020). Artificial intelligence as digital agency. European Journal of Information Systems, 29 (1), 1–8. https://doi.org/10.1080/0960085X.2020.1721947

Alt, R. (2018). Electronic markets and current general research. Electronic Markets, 28 (2), 123–128. https://doi.org/10.1007/s12525-018-0299-0

Alt, R. (2021). Electronic markets on the next convergence. Electronic Markets, 31 (1), 1–9. https://doi.org/10.1007/s12525-021-00471-6

Arnott, D. (2006). Cognitive biases and decision support systems development: a design science approach. Information Systems Journal, 16 (1), 55–78. https://doi.org/10.1111/j.1365-2575.2006.00208.x

Arnott, D., & Pervan, G. (2005). A critical analysis of decision support systems research. Journal of Information Technology, 20 (2), 67–87. https://doi.org/10.1057/palgrave.jit.2000035

Baier, L., Kühl, N., & Satzger, G. (2019). How to cope with change? Preserving validity of predictive services over time. Hawaii International Conference on System Sciences (HICSS-52) . https://doi.org/10.5445/IR/1000085769

Bakos, J. Y., & Treacy, M. E. (1986). Information technology and corporate strategy: a research perspective. MIS Quarterly, 107–119 . https://doi.org/10.2307/249029

Bellman, R. (1978). In Boyd & Fraser. (Ed.), An introduction to artificial intelligence: Can computers think?

Google Scholar  

Berns, K., & Hirth, J. (2006). Control of facial expressions of the humanoid robot head ROMAN. IEEE International Conference on Intelligent Robots and Systems, 3119–3124 . https://doi.org/10.1109/IROS.2006.282331

Billings, D., Davidson, A., Schaeffer, J., & Szafron, D. (2002). The challenge of poker. Artificial Intelligence, 134 (1–2), 201–240. https://doi.org/10.1016/S0004-3702(01)00130-8

Brady, M. (1984). Robotics and artificial intelligence. In M. Brady, L. A. Gerhardt, & H. F. Davidson (Eds.), Artificial intelligence (Vol. 26, Issue 1). Springer. https://doi.org/10.1007/978-3-642-82153-0

Chapter   Google Scholar  

Brink, J. A. (2017). Big data management, access, and protection. Journal of the American College of Radiology, 14 (5), 579–580. https://doi.org/10.1016/j.jacr.2017.03.024

Brynjolfsson, E., & Mcafee, A. (2017). The business of artificial intelligence. Harvard Business Review , 1–20.

Camerer, C. F. (2017). Artificial intelligence and behavioral economics. In Economics of Artificial Intelligence . University of Chicago Press.

Campesato, O. (2020). Artificial intelligence, machine learning, and deep learning . Mercury Learning & Information.

Carnap, R. (1955). Meaning and synonymy in natural languages. Philosophical Studies, 6 (3), 33–47.  https://doi.org/10.1007/BF02330951

Chapman, P., Clinton, J., Kerber, R., Khabaza, T., Reinartz, T., Shearer, C., & Wirth, R. (2000). CRISP-DM 1.0. CRISP-DM Consortium , 76. https://doi.org/10.1109/ICETET.2008.239

Chuang, T.-T., & Yadav, S. B. (1997). An agent-based architecture of an adaptive decision support system. Americas Conference on Information Systems, Indianapolis, IN .

Clark, A., Fox, C., & Lappin, S. (2010). The handbook of computational linguistics and natural language processing (a. Clark, C. Fox, & S. Lappin (eds.)). Wiley-Blackwell. https://doi.org/10.1002/9781444324044

Collins, C., Dennehy, D., Conboy, K., & Mikalef, P. (2021). Artificial intelligence in information systems research: A systematic literature review and research agenda. International Journal of Information Management, 60 , 102383. https://doi.org/10.1016/j.ijinfomgt.2021.102383

Copeland, M. (2016). What’s the difference between artificial intelligence. Machine learning, and deep learning , 29 .  https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/ . Accessed 3 May 2022.

D’Atri, A., Marco, M., & Casalino, N. (2008). Interdisciplinary aspects of information systems studies. The Italian association for information systems . Physica Heidelberg. https://link.springer.com/book/10.1007/978-3-7908-2010-2

Davey, B., & Cope, C. (2008). Requirements elicitation – What’s missing? Issues in Informing Science and Information Technology, 5 , 543–551. https://doi.org/10.28945/1027

Dejoux, C., & Léon, E. (2018). Métamorphose des managers à l’ère du numérique et de l’intelligence artificielle . Pearson.

Dellermann, D., Lipusch, N., Ebel, P., & Leimeister, J. M. (2019). Design principles for a hybrid intelligence decision support system for business model validation. Electronic Markets, 29 (3), 423–441. https://doi.org/10.1007/s12525-018-0309-2

Dunin-Barkowski, W. (2020). Editorial: Toward and beyond human-level AI. Frontiers in Neurorobotics, 14 . https://doi.org/10.3389/fnbot.2020.617446

Engel, C., Ebel, P., & Leimeister, J. M. (2022). Cognitive automation. Electronic Markets, 32 (1), 339–350. https://doi.org/10.1007/s12525-021-00519-7

Fujii, H., & Managi, S. (2018). Trends and priority shifts in artificial intelligence technology invention: A global patent analysis. Economic Analysis and Policy, 58 , 60–69. https://doi.org/10.1016/j.eap.2017.12.006

Fukuda, T., Michelini, R., Potkonjak, V., Tzafestas, S., Valavanis, K., & Vukobratovic, M. (2001). How far away is “artificial man.” IEEE Robotics & Automation Magazine, 8 (1), 66–73. https://doi.org/10.1109/100.924367

Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46 (4), 1–37. https://doi.org/10.1145/2523813

Gao, S., & Xu, D. (2009). Conceptual modeling and development of an intelligent agent-assisted decision support system for anti-money laundering. Expert Systems with Applications, 36 (2), 1493–1504. https://doi.org/10.1016/j.eswa.2007.11.059

Geller, T. (2014). How do you feel? Your computer knows. Communications of the ACM, 6 (8), 24–26. https://doi.org/10.1016/S1364-6613(02)01946-0

Ghavamipoor, H., & Hashemi Golpayegani, S. A. (2020). A reinforcement learning based model for adaptive service quality management in E-commerce websites. Business & Information Systems Engineering, 62 (2), 159–177. https://doi.org/10.1007/s12599-019-00583-6

Grosu, R. (2022). Can artificial intelligence improve our health? In Strategies for sustainability of the earth system (pp. 273–281). Springer. https://doi.org/10.1007/978-3-030-74458-8_17

Guizzo, E. (2014). How Aldebaran robotics built its friendly humanoid robot, pepper . IEEE Spectrum.  https://www.spectrum.ieee.org/how-aldebaran-robotics-built-its-friendly-humanoid-robot-pepper

Hastie, T., Tibshirani, R., & Friedman, J. (2017). The elements of statistical learning: Data mining, inference and prediction (Vol. 9). Springer.

Hatzilygeroudis, I., & Prentzas, J. (2004). Using a hybrid rule-based approach in developing an intelligent tutoring system with knowledge acquisition and update capabilities. Expert Systems with Applications, 26 (4), 477–492. https://doi.org/10.1016/j.eswa.2003.10.007

Haugeland, J. (1989). Artificial intelligence: The very idea . MIT Press.

Book   Google Scholar  

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . https://doi.org/10.1109/CVPR.2016.90

He, S., Rui, H., & Whinston, A. B. (2018). Social media strategies in product-harm crises. Information Systems Research, 29 (2), 362–380. https://doi.org/10.1287/isre.2017.0707

Hegazy, I. M., Faheem, H. M., Al-Arif, T., & Ahmed, T. (2005). Performance evaluation of agent-based IDS. Proceedings of the 2nd international conference on intelligent computing and information systems (ICICIS 2005) (pp. 314–319).

Hein, A., Weking, J., Schreieck, M., Wiesche, M., Böhm, M., & Krcmar, H. (2019). Value co-creation practices in business-to-business platform ecosystems. Electronic Markets, 29 (3), 503–518. https://doi.org/10.1007/s12525-019-00337-y

Hemmer, P., Schemmer, M., Vössing, M., & Kühl, N. (2021). Human-AI complementarity in hybrid intelligence systems: A structured literature review. PACIS 2021 Proceedings .

Hirt, R., Kühl, N., & Satzger, G. (2019). Cognitive computing for customer profiling: meta classification for gender prediction. Electronic Markets, 29 (1), 93–106. https://doi.org/10.1007/s12525-019-00336-z

Hunke, F., Heinz, D., & Satzger, G. (2022). Creating customer value from data: Foundations and archetypes of analytics-based services. Electronic Markets, 32 (2), 1–19. https://doi.org/10.1007/s12525-021-00506-y

ICO. (2017). Big data, artificial intelligence, machine learning and data protection .  https://www.ico.org.uk/media/for-organisations/documents/2013559/big-data-ai-ml-and-data-protection.pdf

James, H., & Paul, R. (2018). Collaborative intelligence: Humans and AI are joining forces (pp. 114–123). Harvard Business Review.

Janiesch, C., Zschech, P., & Heinrich, K. (2021). Machine learning and deep learning. Electronic Markets, 31 (3), 685–695. https://doi.org/10.1007/s12525-021-00475-2

Jordan, M. I., & Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349 (6245), 255–260. https://doi.org/10.1126/science.aaa8415

Jorge, A. M., Leal, J. P., Anand, S. S., & Dias, H. (2014). A study of machine learning methods for detecting user interest during web sessions. Proceedings of the 18th International Database Engineering & Applications Symposium on - IDEAS ‘14 , 149–157. https://doi.org/10.1145/2628194.2628239

Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4 , 237–285. https://doi.org/10.1613/jair.301

Kitts, B., & Leblanc, B. (2004). Optimal bidding on keyword auctions. Electronic Markets, 14 (3), 186–201. https://doi.org/10.1080/1019678042000245119

Kloör, B., Monhof, M., Beverungen, D., & Braäer, S. (2018). Design and evaluation of a model-driven decision support system for repurposing electric vehicle batteries. European Journal of Information Systems, 27 (2), 171–188. https://doi.org/10.1057/s41303-017-0044-3

Koza, J. R., Bennett, F. H., Andre, D., & Keane, M. A. (1996). Automated design of both the topology and sizing of analog electrical circuits using genetic programming. In J. S. Gero, & F.  Sudweeks (Eds.),  Artificial Intelligence in Design ’96 . Springer. https://doi.org/10.1007/978-94-009-0279-4_9

Kühl, N., Hirt, R., Baier, L., Schmitz, B., & Satzger, G. (2021). How to conduct rigorous supervised machine learning in information systems research: The supervised machine learning report card. Communications of the Association for Information Systems, 48 (1), 589–615. https://doi.org/10.17705/1CAIS.04845

Kühl, N., Mühlthaler, M., & Goutier, M. (2020). Supporting customer-oriented marketing with artificial intelligence: Automatically quantifying customer needs from social media. Electronic Markets, 30 (2), 351–367. https://doi.org/10.1007/s12525-019-00351-0

Lange, P. G. (2008). Terminological obfuscation in online research. In Handbook of Research on Computer Mediated Communication (pp. 436–450). IGI Global. https://doi.org/10.4018/978-1-59904-863-5.ch033

Legg, S., & Hutter, M. (2007). Universal intelligence: A definition of machine intelligence. Minds and Machines, 17 (4), 391–444. https://doi.org/10.1007/s11023-007-9079-x

Liebman, E., Saar-Tsechansky, M., & Stone, P. (2015). Dj-mc: A reinforcement-learning agent for music playlist recommendation. Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems , 591–599.

Lieto, A., Bhatt, M., Oltramari, A., & Vernon, D. (2018). The role of cognitive architectures in general artificial intelligence. Cognitive Systems Research, 48 , 1–3. https://doi.org/10.1016/j.cogsys.2017.08.003

McCarthy, J., Minsky, M. L., Rochester, N., & Shannon, C. E. (1956). A proposal for the Dartmouth summer research project on artificial intelligence. Dartmouth Conference. https://doi.org/10.1609/aimag.v27i4.1904

McDermott, D., & Charniak, E. (1985). Introduction to artificial intelligence. International Journal of Adaptive Control and Signal Processing, 2 (2), 148–149.

Mitchell, T. M. (1997). Machine learning. Annual Review Of Computer Science (Issue 1). https://doi.org/10.1145/242224.242229

Mitchell, T. M., Cohen, W., Hruschka, E., Talukdar, P., Betteridge, J., Carlson, A., Mishra, B. D., Gardner, M., Kisiel, B., Krishnamurthy, J., Lao, N., Mazaitis, K., Mohamed, T., Nakashole, N., Platanios, E. A., Ritter, A., Samadi, M., Settles, B., Wang, R., Wijaya, D., Gupta, A., Chen, X., Saparov, A., Greaves, M., & Welling, J. (2015). Never-ending learning. AAAI Conference on Artificial Intelligence , 2302–2310.

Müller, O., Junglas, I., Brocke, J. V., & Debortoli, S. (2016). Utilizing big data analytics for information systems research: Challenges, promises and guidelines. European Journal of Information Systems, 25 (4), 289–302. https://doi.org/10.1057/ejis.2016.2

Nawrocki, T., Maldjian, P. D., Slasky, S. E., & Contractor, S. G. (2018). Artificial intelligence and radiology: Have rumors of the radiologist’s demise been greatly exaggerated? Academic Radiology. https://doi.org/10.1016/j.acra.2017.12.027

Neisser, U. (1967). Cognitive psychology. Thinkingjudgement and Decision Makin . https://doi.org/10.1126/science.198.4319.816

Neuhofer, B., Buhalis, D., & Ladkin, A. (2015). Smart technologies for personalized experiences: a case study in the hospitality domain. Electronic Markets, 25 (3), 243–254.  https://doi.org/10.1007/s12525-015-0182-1

Neuhofer, B., Magnus, B., & Celuch, K. (2021). The impact of artificial intelligence on event experiences: A scenario technique approach. Electronic Markets, 31 (3), 601–617.  https://doi.org/10.1007/s12525-020-00433-4

Newell, A., & Simon, H. A. (1961). GPS, a program that simulates human thought . (Report of the Defense Technical Information Center).  https://www.apps.dtic.mil/sti/citations/AD0294731

Ongsulee, P. (2017). Artificial intelligence, machine learning and deep learning. 2017 15th International Conference on ICT and Knowledge Engineering (ICT\&KE) , 1–6. https://doi.org/10.1109/ICTKE.2017.8259629

Oroszi, F., & Ruhland, J. (2010). An early warning system for hospital acquired. 18th European Conference on Information Systems (ECIS) .  https://www.aisel.aisnet.org/ecis2010/93

Phillips-Wren, G., Power, D. J., & Mora, M. (2019). Cognitive bias, decision styles, and risk attitudes in decision making and DSS . Taylor & Francis. https://doi.org/10.1080/12460125.2019.1646509

Poole, D. L., Mackworth, A., & Goebel, R. G. (1998). Computational intelligence and knowledge. Computational Intelligence: A Logical Approach, Ci , 1–22.

Power, D. J., Cyphert, D., & Roth, R. M. (2019). Analytics, bias, and evidence: The quest for rational decision making. Journal of Decision Systems, 28 (2), 120–137. https://doi.org/10.1080/12460125.2019.1623534

Rai, A., Constantinides, P., & Sarker, S. (2019). Next generation digital platforms: Toward human-AI hybrids. MIS Quarterly, 43 (1), iii–ix.

Rich, E., & Knight, K. (1991). Artificial intelligence . McGraw-Hill.

Ritchie, S. G. (1990). A knowledge-based decision support architecture for advanced traffic management. Transportation Research Part A: General, 24 (1), 27–37. https://doi.org/10.1016/0191-2607(90)90068-H

Rostami, M., Kolouri, S., Kim, K., & Eaton, E. (2017). Multi-agent distributed lifelong learning for collective knowledge acquisition. ArXiv preprint ArXiv:1709.05412 . https://doi.org/10.48550/arXiv.1709.05412

Ruelens, F., Iacovella, S., Claessens, B. J., & Belmans, R. (2015). Learning agent for a heat-pump thermostat with a set-back strategy using model-free reinforcement learning. Energies, 8 (8), 8300–8318. https://doi.org/10.3390/en8088300

Russell, S. J., & Norvig, P. (2020). Artificial intelligence: A modern approach. In Artificial Intelligence (3rd ed.). https://doi.org/10.1017/S0269888900007724

Samtani, S., Chinn, R., Chen, H., & Nunamaker Jr., J. F. (2017). Exploring emerging hacker assets and key hackers for proactive cyber threat intelligence. Journal of Management Information Systems, 34 (4), 1023–1053. https://doi.org/10.1080/07421222.2017.1394049

Schleiffer, R. (2005). An intelligent agent model. European Journal of Operational Research, 166 (3), 666–693. https://doi.org/10.1016/j.ejor.2004.03.039

Schuetz, S., & Venkatesh, V. (2020). Research perspectives: The rise of human machines: How cognitive computing systems challenge assumptions of user-system interaction. Journal of the Association for Information Systems, 21 (2), 460–482. https://doi.org/10.17705/1jais.00608

The Washington Post. (2018, April 10). Transcript of Mark Zuckerberg’s senate hearing. https://www.washingtonpost.com/news/the-switch/wp/2018/04/10/transcript-of-mark-zuckerbergs-senate-hearing/

Thiebes, S., Lins, S., & Sunyaev, A. (2021). Trustworthy artificial intelligence. Electronic Markets, 31 (2), 1–18. https://doi.org/10.1007/s12525-020-00441-4

Turing, A. M. (1950). Computing machine and intelligence. MIND, LIX (236), 433–460. https://doi.org/10.1093/2Fmind/2FLIX.236.433

Ullman, S. (2019). Using neuroscience to develop artificial intelligence. Science, 363 (6428), 692–693. https://doi.org/10.1126/science.aau6595

Wang, H., Kwong, S., Jin, Y., Wei, W., & Man, K.-F. (2005). Agent-based evolutionary approach for interpretable rule-based knowledge extraction. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 35 (2), 143–155. https://doi.org/10.1109/TSMCC.2004.841910

Wang, K., Wang, B., & Peng, L. (2009). CVAP: Validation for cluster analyses. Data Science Journal, 904220071 . https://doi.org/10.2481/dsj.007-020

Waseem, Z., & Hovy, D. (2016). Hateful symbols or hateful people? Predictive features for hate speech detection on twitter. Proceedings of the NAACL Student Research Workshop , 88–93. https://doi.org/10.18653/v1/N16-2013

Witten, I. H., Frank, E., & Hall, M. A. (2011). Data mining: Practical machine learning tools and techniques, third edition. In Annals of physics (Vol. 54, Issue 2).  https://doi.org/10.1002/1521-3773(20010316)40:6<9823::AID-ANIE9823>3.3.CO;2-C

Yu, Y., Eshghi, A., & Lemon, O. (2017). VOILA : An optimised dialogue system for interactively learning visually-grounded word meanings (demonstration system). Proceedings of the SIGDIAL 2017 Conference , 197–200.

Zhai, Z., Martínez, J. F., Beltran, V., & Martínez, N. L. (2020). Decision support systems for agriculture 4.0: Survey and challenges. Computers and Electronics in Agriculture, 170 , 105256. https://doi.org/10.1016/j.compag.2020.105256

Zheng, Q., Wu, Z., Cheng, X., Jiang, L., & Liu, J. (2013). Learning to crawl deep web. Information Systems, 38 (6), 801–819.  https://doi.org/10.1016/j.is.2013.02.001

Zheng, Z., Zheng, L., & Yang, Y. (2017). Pedestrian alignment network for large-scale person re-identification. ArXiv Preprint ArXiv:1707.00408 .

Zhou, Z.-J., Hu, C.-H., Yang, J.-B., Xu, D.-L., & Zhou, D.-H. (2009). Online updating belief rule based system for pipeline leak detection under expert intervention. Expert Systems with Applications, 36 (4), 7700–7709.  https://doi.org/10.1016/j.eswa.2008.09.032

Zhu, X. J. (2005). Semi-supervised learning literature survey . University of Wisconsin-Madison, Department of Computer Sciences. https://www.digital.library.wisc.edu/1793/60444

Download references

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

Karlsruhe Institute of Technology (KIT), Kaiserstr. 89, 76133, Karlsruhe, Germany

Niklas Kühl, Max Schemmer & Gerhard Satzger

Technical University of Darmstadt (TU Darmstadt), Hochschulstr. 1, 64289, Darmstadt, Germany

Marc Goutier

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Niklas Kühl .

Additional information

Responsible Editor: Ioanna Constantiou

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Kühl, N., Schemmer, M., Goutier, M. et al. Artificial intelligence and machine learning. Electron Markets 32 , 2235–2244 (2022). https://doi.org/10.1007/s12525-022-00598-0

Download citation

Received : 16 May 2022

Accepted : 23 September 2022

Published : 09 November 2022

Issue Date : December 2022

DOI : https://doi.org/10.1007/s12525-022-00598-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

JEL classification

• Find a journal
• Publish with us
• Track your research

Help | Advanced Search

Computer Science > Machine Learning

Title: mer 2024: semi-supervised learning, noise robustness, and open-vocabulary multimodal emotion recognition.

Abstract: Multimodal emotion recognition is an important research topic in artificial intelligence. Over the past few decades, researchers have made remarkable progress by increasing dataset size and building more effective architectures. However, due to various reasons (such as complex environments and inaccurate labels), current systems still cannot meet the demands of practical applications. Therefore, we plan to organize a series of challenges around emotion recognition to further promote the development of this field. Last year, we launched MER2023, focusing on three topics: multi-label learning, noise robustness, and semi-supervised learning. This year, we continue to organize MER2024. In addition to expanding the dataset size, we introduce a new track around open-vocabulary emotion recognition. The main consideration for this track is that existing datasets often fix the label space and use majority voting to enhance annotator consistency, but this process may limit the model's ability to describe subtle emotions. In this track, we encourage participants to generate any number of labels in any category, aiming to describe the emotional state as accurately as possible. Our baseline is based on MERTools and the code is available at: this https URL .

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Artificial intelligence and machine learning in spine research

Affiliation.

• 1 Laboratory of Biological Structures Mechanics IRCCS Istituto Ortopedico Galeazzi Milan Italy.
• PMID: 31463458
• PMCID: PMC6686793
• DOI: 10.1002/jsp2.1044

Artificial intelligence (AI) and machine learning (ML) techniques are revolutionizing several industrial and research fields like computer vision, autonomous driving, natural language processing, and speech recognition. These novel tools are already having a major impact in radiology, diagnostics, and many other fields in which the availability of automated solution may benefit the accuracy and repeatability of the execution of critical tasks. In this narrative review, we first present a brief description of the various techniques that are being developed nowadays, with special focus on those used in spine research. Then, we describe the applications of AI and ML to problems related to the spine which have been published so far, including the localization of vertebrae and discs in radiological images, image segmentation, computer-aided diagnosis, prediction of clinical outcomes and complications, decision support systems, content-based image retrieval, biomechanics, and motion analysis. Finally, we briefly discuss major ethical issues related to the use of AI in healthcare, namely, accountability, risk of biased decisions as well as data privacy and security, which are nowadays being debated in the scientific community and by regulatory agencies.

Keywords: artificial neural networks; deep learning; ethical implications; outcome prediction; segmentation.

Publication types

NTRS - NASA Technical Reports Server

Available downloads, related records.

This paper is in the following e-collection/theme issue:

Published on 29.4.2024 in Vol 26 (2024)

The Applications of Artificial Intelligence for Assessing Fall Risk: Systematic Review

Authors of this article:

• Ana González-Castro 1 , PT, MSc   ; 
• Raquel Leirós-Rodríguez 2 , PT, PhD   ; 
• Camino Prada-García 3 , MD, PhD   ; 
• José Alberto Benítez-Andrades 4 , PhD  

1 Nursing and Physical Therapy Department, Universidad de León, Ponferrada, Spain

2 SALBIS Research Group, Nursing and Physical Therapy Department, Universidad de León, Ponferrada, Spain

3 Department of Preventive Medicine and Public Health, Universidad de Valladolid, Valladolid, Spain

4 SALBIS Research Group, Department of Electric, Systems and Automatics Engineering, Universidad de León, León, Spain

Corresponding Author:

Ana González-Castro, PT, MSc

Nursing and Physical Therapy Department

Universidad de León

Astorga Ave

Ponferrada, 24401

Phone: 34 987442000

Email: [email protected]

Background: Falls and their consequences are a serious public health problem worldwide. Each year, 37.3 million falls requiring medical attention occur. Therefore, the analysis of fall risk is of great importance for prevention. Artificial intelligence (AI) represents an innovative tool for creating predictive statistical models of fall risk through data analysis.

Objective: The aim of this review was to analyze the available evidence on the applications of AI in the analysis of data related to postural control and fall risk.

Methods: A literature search was conducted in 6 databases with the following inclusion criteria: the articles had to be published within the last 5 years (from 2018 to 2024), they had to apply some method of AI, AI analyses had to be applied to data from samples consisting of humans, and the analyzed sample had to consist of individuals with independent walking with or without the assistance of external orthopedic devices.

Results: We obtained a total of 3858 articles, of which 22 were finally selected. Data extraction for subsequent analysis varied in the different studies: 82% (18/22) of them extracted data through tests or functional assessments, and the remaining 18% (4/22) of them extracted through existing medical records. Different AI techniques were used throughout the articles. All the research included in the review obtained accuracy values of >70% in the predictive models obtained through AI.

Conclusions: The use of AI proves to be a valuable tool for creating predictive models of fall risk. The use of this tool could have a significant socioeconomic impact as it enables the development of low-cost predictive models with a high level of accuracy.

Trial Registration: PROSPERO CRD42023443277; https://tinyurl.com/4sb72ssv

Introduction

According to alarming figures reported by the World Health Organization in 2021, falls cause 37.3 million injuries annually that require medical attention and result in 684,000 deaths [ 1 ]. These figures indicate a significant impact of falls on the health care system and on society, both directly and indirectly [ 2 , 3 ].

Life expectancy has progressively increased over the years, leading to an aging population [ 4 ]. By 2050, it is estimated that 16% of the population will be >65 years of age. In this group, the incidence of falls has steadily risen, becoming the leading cause of accidental injury and death (accounting for 55.8% of such deaths, according to some research) [ 5 , 6 ]. It is estimated that 30% of this population falls at least once a year, negatively impacting their physical and psychological well-being [ 7 , 8 ].

Physically, falls are often associated with severe complications that can lead to extended hospitalizations [ 9 ]. These hospitalizations are usually due to serious injuries, often cranioencephalic trauma, fractures, or soft tissue injuries [ 10 , 11 ]. Psychologically, falls among the older adult population tend to result in self-imposed limitations due to the fear of falling again [ 10 , 12 ]. These limitations lead to social isolation as individuals avoid participating in activities or even individual mobility [ 13 ]. Consequently, falls can lead to psychological conditions such as anxiety and depression [ 14 , 15 ]. Numerous research studies on the risk of falls are currently underway, with ongoing investigations into various innovations and intervention ideas [ 16 - 19 ]. These studies encompass the identification of fall risk factors [ 20 , 21 ], strategies for prevention [ 22 , 23 ], and the outcomes following rehabilitation [ 23 , 24 ].

In the health care field, artificial intelligence (AI) is characterized by data management and processing, offering new possibilities to the health care paradigm [ 24 ]. Some applications of AI in the health care domain include assessing tumor interaction processes [ 25 ], serving as a tool for image-based diagnostics [ 26 , 27 ], participating in virus detection [ 28 ], and, most importantly, as a statistical and predictive method [ 29 - 32 ].

Several publications have combined AI techniques to address health care issues [ 33 - 35 ]. Within the field of predictive models, it is important to understand certain differentiations. In AI, we have machine learning and deep learning [ 36 - 38 ]. Machine learning encompasses a set of techniques applied to data and can be done in a supervised or unsupervised manner [ 39 , 40 ]. On the other hand, deep learning is typically used to work with larger data sets compared to machine learning, and its computational cost is higher [ 41 , 42 ].

Some examples of AI techniques include the gradient boosting machine [ 43 ], learning method, and the long short-term memory (LSTM) [ 44 ] and the convolutional neural network (CNN) [ 45 ], all of them are deep learning methods.

For all the reasons mentioned in the preceding section, it was considered necessary to conduct a systematic review to analyze the scientific evidence of AI applications in the analysis of data related to postural control and the risk of falls.

Data Sources and Searches

This systematic review and meta-analysis were prospectively registered on PROSPERO (ID CRD42023443277) and followed the Meta-Analyses of Observational Studies in Epidemiology checklist [ 46 ] and the recommendations of the Cochrane Collaboration [ 47 ].

The search was conducted in January 2024 on the following databases: PubMed, Scopus, ScienceDirect, Web of Science, CINAHL, and Cochrane Library. The Medical Subject Headings (MeSH) terms used for the search included machine learning , artificial intelligent , accidental falls , rehabilitation , and physical therapy specialty . The terms “predictive model” and “algorithms” were also used. These terms were combined using the Boolean operators AND and OR ( Textbox 1 ).

• (“machine learning”[MeSH Terms] OR “artificial intelligent”[MeSH Terms]) AND “accidental falls”[MeSH Terms]
• (“machine learning”[MeSH Terms] OR “artificial intelligent”) AND (“rehabilitation”[MeSH Terms] OR “physical therapy specialty”[MeSH Terms])
• “accidental falls” [Title/Abstract] AND “algorithms” [Title/Abstract]
• “accidental falls”[Title/Abstract] AND “predictive model” [Title/Abstract]
• TITLE-ABS-KEY (“machine learning” OR “artificial intelligent”) AND TITLE-ABS-KEY (“accidental falls”)
• TITLE-ABS-KEY (“machine learning” OR “artificial intelligent”) AND TITLE-ABS-KEY (“rehabilitation” OR “physical therapy specialty”)
• TITLE-ABS-KEY (“accidental falls” AND “algorithms”)
• TITLE-ABS-KEY (“accidental falls” AND “predictive model”)

ScienceDirect

• Title, abstract, keywords: (“machine learning” OR “artificial intelligent”) AND “accidental falls”
• Title, abstract, keywords: (“machine learning” OR “artificial intelligent”) AND (“rehabilitation” OR “physical therapy specialty”)
• Title, abstract, keywords: (“accidental falls” AND “algorithms”)
• Title, abstract, keywords: (“accidental falls” AND “predictive model”)

Web of Science

• TS=(“machine learning” OR “artificial intelligent”) AND TS=“accidental falls”
• TS=(“machine learning” OR “artificial intelligent”) AND TS= (“rehabilitation” OR “physical therapy specialty”)
• AB= (“accidental falls” AND “algorithms”)
• AB= (“accidental falls” AND “predictive model”)
• (MH “machine learning” OR MH “artificial intelligent”) AND MH “accidental falls”
• (MH “machine learning” OR MH “artificial intelligent”) AND (MH “rehabilitation” OR MH “physical therapy specialty”)
• (AB “accidental falls”) AND (AB “algorithms”)
• (AB “accidental falls”) AND (AB “predictive model”)

Cochrane Library

• (“machine learning” OR “artificial intelligent”) in Title Abstract Keyword AND “accidental falls” in Title Abstract Keyword
• (“machine learning” OR “artificial intelligent”) in Title Abstract Keyword AND (“rehabilitation” OR “physical therapy specialty”) in Title Abstract Keyword
• “accidental falls” in Title Abstract Keyword AND “algorithms” in Title Abstract Keyword
• “accidental falls” in Title Abstract Keyword AND “predictive model” in Title Abstract Keyword

Study Selection

After removing duplicates, 2 reviewers (AGC and RLR) independently screened articles for eligibility. In the case of disagreement, a third reviewer (JABA) finally decided whether the study should be included or not. We calculated the κ coefficient and percentage agreement scores to assess reliability before any consensus and estimated the interrater reliability using κ. Interrater reliability was estimated using κ>0.7 indicating a high level of agreement between the reviewers, κ of 0.5 to 0.7 indicating a moderate level of agreement, and κ<0.5 indicating a low level of agreement [ 48 ].

For the selection of results, the inclusion criteria were established as follows: (1) articles should have been published in the last 5 years (from 2018 to the present); (2) they must apply some AI method; (3) AI analyses should be applied to data from samples of humans; and (4) the sample analyzed should consist of people with independent walking, with or without the use of external orthopedic devices.

After screening the data, extracting, obtaining, and screening the titles and abstracts for inclusion criteria, the selected abstracts were obtained in full texts. Titles and abstracts lacking sufficient information regarding inclusion criteria were also obtained as full texts. Full-text articles were selected in case of compliance with inclusion criteria by the 2 reviewers using a data extraction form.

Data Extraction and Quality Assessment

The 2 reviewers mentioned independently extracting data from the included studies using a customized data extraction table in Excel (Microsoft Corporation). In case of disagreement, both reviewers debated until an agreement was reached.

The data extracted from the included articles for further analysis were: demographic information (title, authors, journal, and year), characteristics of the sample (age, inclusion and exclusion criteria, and number of participants), study-specific parameters (study type, AI techniques applied, and data analyzed), and the results obtained. Tables were used to describe both the studies’ characteristics and the extracted data.

Assessment of Risk of Bias

The methodological quality of the selected articles was evaluated using the Critical Review Form for Quantitative Studies [ 49 ]. The ROBINS-E (Risk of Bias in Nonrandomized Studies of Exposures) tool was used to evaluate the risk of bias [ 50 ].

Characteristics of the Selected Studies

A total of 3858 articles were initially retrieved, with 1563 duplicates removed. From the remaining 2295 articles, 2271 were excluded based on the initial selection criteria, leaving 24 articles for the subsequent analysis. In this second analysis, 2 articles were removed as they were systematic reviews, and 22 articles were finally selected [ 51 - 72 ] ( Figure 1 ). After the first reading of all candidate full texts, the kappa score for inclusion of the results of reviewers 1 and 2 was 0.98, indicating a very high level of agreement.

The methodological quality of the 22 analyzed studies (Table S1 in Multimedia Appendix 1 [ 51 , 52 , 54 , 56 , 58 , 59 , 61 , 63 , 64 , 69 , 70 , 72 ]) ranged from 11 points in 2 (9.1%) studies [ 52 , 65 ] to 16 points in 7 (32%) studies [ 53 , 54 , 56 , 63 , 69 - 71 ].

Study Characteristics and Risk of Bias

All the selected articles were cross-sectional observational studies ( Table 1 ).

In total, 34 characteristics affecting the risk of falls were extracted and classified into high fall-risk and low fall-risk groups with the largest sample sizes significantly differing from the rest. Studies based on data collected from various health care systems had larger sample sizes, ranging from 22,515 to 265,225 participants [ 60 , 65 , 67 ]. In contrast, studies that applied some form of evaluation test had sample sizes ranging from 8 participants [ 56 ] to 746 participants [ 55 ].

It is worth noting the various studies conducted by Dubois et al [ 54 , 72 ], whose publications on fall risk and machine learning started in 2018 and progressed until 2021. A total of 9.1% (2/22) of the articles by this author were included in the final selection [ 54 , 72 ]. Both articles used samples with the same characteristics, even though the first one was composed of 43 participants [ 54 ] and the last one had 30 participants [ 72 ]. All 86.4% (19/22) of the articles used samples of individuals aged ≥65 years [ 51 - 60 , 62 - 65 , 68 - 72 ]. In the remaining 13.6% (3/22) of the articles, the ages ranged between 16 and 62 years [ 61 , 66 , 67 ].

Althobaiti et al [ 61 ] used a sample of participants between the ages of 19 and 35 years for their research, where these participants had to reproduce examples of falls for subsequent analysis. In 2022, Ladios-Martin et al [ 67 ] extracted medical data from participants aged >16 years for their research. Finally, in 2023, the study by Maray et al [ 66 ] used 3 types of samples, with ages ranging from 21 to 62 years. Among the 22 selected articles, only 1 (4.5%) of them did not describe the characteristics of its sample [ 52 ].

Finally, regarding the sex of the samples, 13.6% (3/22) of the articles specified in the characteristics of their samples that only female individuals were included among their participants [ 53 , 59 , 70 ].

a AI: artificial intelligence.

b ML: machine learning.

c nd: none described.

d ADL: activities of daily living.

e TUG: Timed Up and Go.

f BBS: Berg Balance Scale.

g ASM: associative skill memories.

h CNN: convolutional neural network.

i FP: fall prevention.

j IMU: inertial measurement unit.

k AUROC: area under the receiver operating characteristic curve.

l AUPR: area under the precision-recall curve.

m MFS: Morse Fall Scale.

n XGB: extreme gradient boosting.

o MCT: motor control test.

p GBM: gradient boosting machine.

q RF: random forest.

r LOOCV: leave-one-out cross-validation.

s LSTM: long short-term memory.

Applied Assessment Procedures

All articles initially analyzed the characteristics of their samples to subsequently create a predictive model of the risk of falls. However, they did not all follow the same evaluation process.

Regarding the applied assessment procedures, 3 main options stood out: studies with tests or assessments accompanied by sensors or accelerometers [ 51 - 57 , 59 , 61 - 63 , 66 , 70 - 72 ], studies with tests or assessments accompanied by cameras [ 68 , 69 ], or studies based on medical records [ 58 , 60 , 65 , 67 ] ( Figure 2 ). Guillan et al [ 64 ] performed a physical and functional evaluation of the participants. In their study, they evaluated parameters such as walking speed, stride frequency and length, and the minimum space between the toes. Afterward, they asked them to record the fall events they had during the past 2 years in a personal diary.

In total, 22.7% (5/22) of the studies used the Timed Up and Go test [ 53 , 54 , 69 , 71 , 72 ]. In 18.2% (4/22) of them, the participants performed the test while wearing a sensor to collect data [ 53 , 54 , 71 , 72 ]. In 1 (4.5%) study, the test was recorded with a camera for later analysis [ 69 ]. Another commonly used method in studies was to ask participants to perform everyday tasks or activities of daily living while a sensor collected data for subsequent analysis. Specifically, 18.2% (4/22) of the studies used this method to gather data [ 51 , 56 , 61 , 62 ].

A total of 22.7% (5/22) of the studies asked participants to simulate falls and nonfalls while a sensor collected data [ 52 , 61 - 63 , 66 ]. In this way, the data obtained were used to create the predictive model of falls. As for the tests used, Eichler et al [ 68 ] asked participants to perform the Berg Balance Scale while a camera recorded their performance.

Finally, other authors created their own battery of tests for data extraction [ 55 , 59 , 64 , 70 ]. Gillain et al [ 64 ] used gait records to analyze speed, stride length, frequency, symmetry, regularity, and foot separation. Hu et al [ 59 ] asked their participants to perform normal walking, the postural reflexive response test, and the motor control test. In the study by Noh et al [ 55 ], gait tests were conducted, involving walking 20 m at different speeds. Finally, Greene et al [ 70 ] created a 12-question questionnaire and asked their participants to maintain balance while holding a mobile phone in their hand.

AI Techniques

The selected articles used various techniques within AI. They all had the same objective in applying these techniques, which was to achieve a predictive and classification model for the risk of falls [ 51 - 72 ].

In chronological order, in 2018, Nait Aicha et al [ 51 ] compared single-task learning models with multitask learning, obtaining better evaluation results through multitask learning. In the same year, Dubois et al [ 54 ] applied AI techniques that analyzed multiple parameters to classify the risk of falls in their sample. Qiu et al [ 53 ], also in the same year, used 6 machine learning models (logistic regression, naïve Bayes, decision tree, RF, boosted tree, and support vector machine) in their research.

In contrast, in 2019, Ferrete et al [ 52 ] compared the applicability of 2 different deep learning models: the classifier based on associative skill memories and a CNN classifier. In the same year, after confirming the applicability of AI as a predictive method for the risk of falls, various authors investigated through methods such as the RF to identify factors that can predict and quantify the risk of falls [ 63 , 65 ].

Among the selected articles, 5 (22.7%) were published in 2020 [ 58 - 62 ]. The research conducted by Tunca et al [ 62 ] compared the applicability of deep learning LSTM networks with traditional machine learning applied to the risk of falls. Hu et al [ 59 ] first used cross-validation, where algorithms were trained randomly, and then used the gradient boosting machine algorithm to classify participants as high or low risk. Ye et al [ 60 ] and Hsu et al [ 58 ] both used the extreme gradient boosting (XGBoost) algorithm based on machine learning to create their predictive model. In the same year, Althobaiti et al [ 61 ] trained machine learning models for their research.

In 2021, Lockhart et al [ 57 ] started using 3 deep learning techniques simultaneously with the same goal as before: to create a predictive model for the risk of falls. Specifically, they used the RF, RF with feature engineering, and RF with feature engineering and linear and nonlinear variables. Noh et al [ 55 ], again in the same year, used the XGBoost algorithm, while Roshdibenam et al [ 71 ], on the other hand, used the CNN algorithm for each location of the wearable sensors used in their research. Various machine learning techniques were used for classifying the risk of falls and for balance loss events in the research by Hauth et al [ 56 ]. Dubois et al [ 72 ] used the following algorithms: decision tree, adaptive boosting, neural net, naïve Bayes, k-nearest neighbors, linear support vector machine, radial basis function support vector machine, RF, and quadratic discriminant analysis. Hauth et al [ 56 ], on the other hand, used regularized logistic regression and bidirectional LSTM networks. In the research conducted by Greene et al [ 70 ], AI was used, but the specific procedure that they followed is not described.

Tang et al [ 69 ] published their research with innovation up to that point. In their study, they used a smart gait analyzer with the help of deep learning techniques to assess the diagnostic accuracy of fall risk through vision. Months later, in August 2022, Ladios-Martin et al [ 67 ] published their research, in which they compared 2 deep learning models to achieve the best results in terms of specificity and sensitivity in detecting fall risk. The first model used the Bayesian Point Machine algorithm with a fall prevention variable, and the second one did not use the variable. They obtained better results when using that variable, a mitigating factor defined as a set of care interventions carried out by professionals to prevent the patient from experiencing a fall during hospitalization. Particularly controversial, as its exclusion could obscure the model’s performance. Eichler et al [ 68 ], on the other hand, used machine learning–based classifier training and later tested the performance of RFs in score predictions.

Finally, in January 2023, Maray et al [ 66 ] published their research, linking the previously mentioned terms (AI and fall risk) with 3 wearable devices that are commonly used today. They collected data through these devices and applied transfer learning to generalize the model across heterogeneous devices.

The results of the 22 articles provided promising data, and all of them agreed on the feasibility of applying various AI techniques as a method for predicting and classifying the risk of falls. Specifically, the accuracy values obtained in the studies exceed 70%. Noh et al [ 55 ] achieved the “lowest” accuracy among the studies conducted, with a 70% accuracy rate. Ribeiro et al [ 52 ] obtained an accuracy of 92.7% when using CNN to differentiate between normal gait and fall events. Hsu et al [ 58 ] further demonstrated that the XGBoost model is more sensitive than the Morse Fall Scale. Similarly, in their comparative study, Nait Aicha et al [ 51 ] also showed that a predictive model created from accelerometer data with AI is comparable to conventional models for assessing the risk of falls. More specifically, Dubois et al [ 54 ] concluded that using 1 gait-related parameter (excluding velocity) in combination with another parameter related to seated position allowed for the correct classification of individuals according to their risk of falls.

Principal Findings

The aim of this research was to analyze the scientific evidence regarding the applications of AI in the analysis of data related to postural control and the risk of falls. On the basis of the analysis of results, it can be asserted that the following risk factors were identified in the analyzed studies: age [ 65 ], daily habits [ 65 ], clinical diagnoses [ 65 ], environmental and hygiene factors [ 65 ], sex [ 64 ], stride length [ 55 , 72 ], gait speed [ 55 ], and posture [ 55 ]. This aligns with other research that also identifies sex [ 73 , 74 ], age [ 73 ], and gait speed [ 75 ].

On the other hand, the “fear of falling” has been identified in various studies as a risk factor and a predictor of falls [ 73 , 76 ], but it was not identified in any of the studies included in this review.

As for the characteristics of the analyzed samples, only 9.1% (2/22) of the articles used a sample composed exclusively of women [ 53 , 59 ], and no article used a sample composed exclusively of men. This fact is incongruent with reality, as women have a longer life expectancy than men, and therefore, the number of women aged >65 years is greater than the number of men of the same age [ 77 ]. Furthermore, women experience more falls than men [ 78 ]. The connection between menopause and its consequences, including osteopenia, suggests a higher risk of falls among older women than among men of the same age [ 79 , 80 ].

Within the realm of analysis tools, the most frequently used devices to analyze participants were accelerometers [ 51 - 57 , 59 , 61 - 63 , 66 , 70 - 72 ]. However, only 36.4% (8/22) of the studies provided all the information regarding the characteristics of these devices [ 51 , 53 , 59 , 61 , 63 , 66 , 70 , 72 ]. On the other hand, 18.2% (4/22) of the studies used the term “inertial measurement unit” as the sole description of the devices used [ 55 - 57 , 71 ].

The fact that most of the analyzed procedures involved the use of inertial sensors reflects the current widespread use of these devices for postural control analysis. These sensors, in general (and triaxial accelerometers in particular), have demonstrated great diagnostic capacity for balance [ 81 ]. In addition, they exhibit good sensitivity and reliability, combined with their portability and low economic cost [ 82 ]. Another advantage of triaxial accelerometers is their versatility in both adult and pediatric populations [ 83 - 86 ], although the studies included in this review did not address the pediatric population.

The remaining studies extracted data from cameras [ 68 , 69 ], medical records [ 58 , 60 , 65 , 67 ], and other functional and clinical tests [ 59 , 64 , 70 ]. Regarding the AI techniques used, out of the 18.2% (4/22) of articles that used deep learning techniques [ 52 , 57 , 62 , 71 ], only 4.5% (1/22) did not provide a description of the sample characteristics [ 52 ]. In this case, the authors focused on the AI landscape, while the rest of the articles strike a balance between AI and health sciences.

Regarding the validity of the generated models, only 40.9% (9/22) of the articles assessed this characteristic [ 52 , 53 , 55 , 61 - 64 , 68 , 69 ]. The authors of these 9 (N=22, 40.9%) articles evaluated the validity of the models through accuracy. All the results obtained reflected accuracies exceeding 70%, with Ribeiro et al [ 52 ] achieving a notable accuracy of 92.7% and 100%. Specifically, they obtained a 92.7% accuracy through the CNN model for distinguishing normal gait, the prefall condition, and the falling situation, considering the step before the fall, and 100% when not considering it [ 52 ].

The positive results of sensitivity and specificity can only be compared between the studies of Qiu et al [ 53 ] and Gillain et al [ 64 ], as they were the only ones to take them into account, and in both investigations, they were very high. Similarly, in the case of the F 1 -score, only Althobaiti et al [ 61 ] examined this validity measure. This measure is the result of combining precision and recall into a single figure, and the outcome obtained by these researchers was promising.

Despite these differences, the 22 studies obtained promising results in the health care field [ 51 - 72 ]. Specifically, their outcomes highlight the potential of AI integration into clinical settings. However, further research is necessary to explore how health care professionals can effectively use these predictive models. Consequently, future research should focus on studying the application and integration of the already-developed models. In this context, fall prevention plans could be implemented for the target populations identified by the predictive models. This approach would allow for a retrospective analysis to determine if the combination of predictive models with prevention programs effectively reduces the prevalence of falls in the population.

Limitations

Regarding limitations, the articles showed significant variation in the sample sizes selected. Moreover, even in the study with the largest sample size (with 265,225 participants [ 60 ]), the amount of data analyzed was relatively small. In addition, several of the databases used were not generated specifically for the published research but rather derived from existing medical records [ 58 , 60 , 65 , 67 ]. This could explain the significant variability in the variables analyzed across different studies.

Despite the limitations, this research has strengths, such as being the first systematic review on the use of AI as a tool to analyze postural control and the risk of falls. Furthermore, a total of 6 databases were used for the literature search, and a comprehensive article selection process was carried out by 3 researchers. Finally, only cross-sectional observational studies were selected, and they shared the same objective.

Conclusions

The use of AI in the analysis of data related to postural control and the risk of falls proves to be a valuable tool for creating predictive models of fall risk. It has been identified that most AI studies analyze accelerometer data from sensors, with triaxial accelerometers being the most frequently used.

For future research, it would be beneficial to provide more detailed descriptions of the measurement procedures and the AI techniques used. In addition, exploring larger databases could lead to the development of more robust models.

Conflicts of Interest

None declared.

Quality scores of reviewed studies (Critical Review Form for Quantitative Studies tool results).

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist.

• Step safely: strategies for preventing and managing falls across the life-course. World Health Organization. 2021. URL: https://www.who.int/publications/i/item/978924002191-4 [accessed 2024-04-02]
• Keall MD, Pierse N, Howden-Chapman P, Guria J, Cunningham CW, Baker MG. Cost-benefit analysis of fall injuries prevented by a programme of home modifications: a cluster randomised controlled trial. Inj Prev. Feb 2017;23(1):22-26. [ CrossRef ] [ Medline ]
• Almada M, Brochado P, Portela D, Midão L, Costa E. Prevalence of falls and associated factors among community-dwelling older adults: a cross-sectional study. J Frailty Aging. 2021;10(1):10-16. [ CrossRef ] [ Medline ]
• Menéndez-González L, Izaguirre-Riesgo A, Tranche-Iparraguirre S, Montero-Rodríguez Á, Orts-Cortés MI. [Prevalence and associated factors of frailty in adults over 70 years in the community]. Aten Primaria. Dec 2021;53(10):102128. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guirguis-Blake JM, Michael YL, Perdue LA, Coppola EL, Beil TL. Interventions to prevent falls in older adults: updated evidence report and systematic review for the US preventive services task force. JAMA. Apr 24, 2018;319(16):1705-1716. [ CrossRef ] [ Medline ]
• Pereira CB, Kanashiro AM. Falls in older adults: a practical approach. Arq Neuropsiquiatr. May 2022;80(5 Suppl 1):313-323. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Byun M, Kim J, Kim M. Physical and psychological factors affecting falls in older patients with arthritis. Int J Environ Res Public Health. Feb 09, 2020;17(3):1098. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Goh HT, Nadarajah M, Hamzah NB, Varadan P, Tan MP. Falls and fear of falling after stroke: a case-control study. PM R. Dec 04, 2016;8(12):1173-1180. [ CrossRef ] [ Medline ]
• Alanazi FK, Lapkin S, Molloy L, Sim J. The impact of safety culture, quality of care, missed care and nurse staffing on patient falls: a multisource association study. J Clin Nurs. Oct 12, 2023;32(19-20):7260-7272. [ CrossRef ] [ Medline ]
• Hossain A, Lall R, Ji C, Bruce J, Underwood M, Lamb SE. Comparison of different statistical models for the analysis of fracture events: findings from the Prevention of Falls Injury Trial (PreFIT). BMC Med Res Methodol. Oct 02, 2023;23(1):216. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Williams CT, Whyman J, Loewenthal J, Chahal K. Managing geriatric patients with falls and fractures. Orthop Clin North Am. Jul 2023;54(3S):e1-12. [ CrossRef ] [ Medline ]
• Gadhvi C, Bean D, Rice D. A systematic review of fear of falling and related constructs after hip fracture: prevalence, measurement, associations with physical function, and interventions. BMC Geriatr. Jun 23, 2023;23(1):385. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lohman MC, Fallahi A, Mishio Bawa E, Wei J, Merchant AT. Social mediators of the association between depression and falls among older adults. J Aging Health. Aug 12, 2023;35(7-8):593-603. [ CrossRef ] [ Medline ]
• Smith AD, Silva AO, Rodrigues RA, Moreira MA, Nogueira JD, Tura LF. Assessment of risk of falls in elderly living at home. Rev Lat Am Enfermagem. Apr 06, 2017;25:e2754. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Koh V, Matchar DB, Chan A. Physical strength and mental health mediate the association between pain and falls (recurrent and/or injurious) among community-dwelling older adults in Singapore. Arch Gerontol Geriatr. Sep 2023;112:105015. [ CrossRef ] [ Medline ]
• Soh SE, Morgan PE, Hopmans R, Barker AL, Ackerman IN. The feasibility and acceptability of a falls prevention e-learning program for physiotherapists. Physiother Theory Pract. Mar 18, 2023;39(3):631-640. [ CrossRef ] [ Medline ]
• Morat T, Snyders M, Kroeber P, De Luca A, Squeri V, Hochheim M, et al. Evaluation of a novel technology-supported fall prevention intervention - study protocol of a multi-centre randomised controlled trial in older adults at increased risk of falls. BMC Geriatr. Feb 18, 2023;23(1):103. [ FREE Full text ] [ CrossRef ] [ Medline ]
• You T, Koren Y, Butts WJ, Moraes CA, Yeh GY, Wayne PM, et al. Pilot studies of recruitment and feasibility of remote Tai Chi in racially diverse older adults with multisite pain. Contemp Clin Trials. May 2023;128:107164. [ CrossRef ] [ Medline ]
• Aldana-Benítez D, Caicedo-Pareja MJ, Sánchez DP, Ordoñez-Mora LT. Dance as a neurorehabilitation strategy: a systematic review. J Bodyw Mov Ther. Jul 2023;35:348-363. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jawad A, Baattaiah BA, Alharbi MD, Chevidikunnan MF, Khan F. Factors contributing to falls in people with multiple sclerosis: the exploration of the moderation and mediation effects. Mult Scler Relat Disord. Aug 2023;76:104838. [ CrossRef ] [ Medline ]
• Warren C, Rizo E, Decker E, Hasse A. A comprehensive analysis of risk factors associated with inpatient falls. J Patient Saf. Oct 01, 2023;19(6):396-402. [ CrossRef ] [ Medline ]
• Gross M, Roigk P, Schoene D, Ritter Y, Pauly P, Becker C, et al. Bundesinitiative Sturzprävention. [Update of the recommendations of the federal falls prevention initiative-identification and prevention of the risk of falling in older people living at home]. Z Gerontol Geriatr. Oct 11, 2023;56(6):448-457. [ CrossRef ] [ Medline ]
• Li S, Li Y, Liang Q, Yang WJ, Zi R, Wu X, et al. Effects of tele-exercise rehabilitation intervention on women at high risk of osteoporotic fractures: study protocol for a randomised controlled trial. BMJ Open. Nov 07, 2022;12(11):e064328. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. Dec 2017;2(4):230-243. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ye Y, Wu X, Wang H, Ye H, Zhao K, Yao S, et al. Artificial intelligence-assisted analysis for tumor-immune interaction within the invasive margin of colorectal cancer. Ann Med. Dec 2023;55(1):2215541. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kuwahara T, Hara K, Mizuno N, Haba S, Okuno N, Fukui T, et al. Current status of artificial intelligence analysis for the treatment of pancreaticobiliary diseases using endoscopic ultrasonography and endoscopic retrograde cholangiopancreatography. DEN Open. Apr 30, 2024;4(1):e267. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yokote A, Umeno J, Kawasaki K, Fujioka S, Fuyuno Y, Matsuno Y, et al. Small bowel capsule endoscopy examination and open access database with artificial intelligence: the SEE-artificial intelligence project. DEN Open. Apr 22, 2024;4(1):e258. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ramalingam M, Jaisankar A, Cheng L, Krishnan S, Lan L, Hassan A, et al. Impact of nanotechnology on conventional and artificial intelligence-based biosensing strategies for the detection of viruses. Discov Nano. Dec 01, 2023;18(1):58. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yerukala Sathipati S, Tsai MJ, Shukla SK, Ho SY. Artificial intelligence-driven pan-cancer analysis reveals miRNA signatures for cancer stage prediction. HGG Adv. Jul 13, 2023;4(3):100190. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Liu J, Dan W, Liu X, Zhong X, Chen C, He Q, et al. Development and validation of predictive model based on deep learning method for classification of dyslipidemia in Chinese medicine. Health Inf Sci Syst. Dec 06, 2023;11(1):21. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Carou-Senra P, Ong JJ, Castro BM, Seoane-Viaño I, Rodríguez-Pombo L, Cabalar P, et al. Predicting pharmaceutical inkjet printing outcomes using machine learning. Int J Pharm X. Dec 2023;5:100181. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Li X, Zhu Y, Zhao W, Shi R, Wang Z, Pan H, et al. Machine learning algorithm to predict the in-hospital mortality in critically ill patients with chronic kidney disease. Ren Fail. Dec 2023;45(1):2212790. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bonnin M, Müller-Fouarge F, Estienne T, Bekadar S, Pouchy C, Ait Si Selmi T. Artificial intelligence radiographic analysis tool for total knee arthroplasty. J Arthroplasty. Jul 2023;38(7 Suppl 2):S199-207.e2. [ CrossRef ] [ Medline ]
• Kao DP. Intelligent artificial intelligence: present considerations and future implications of machine learning applied to electrocardiogram interpretation. Circ Cardiovasc Qual Outcomes. Sep 2019;12(9):e006021. [ FREE Full text ] [ CrossRef ] [ Medline ]
• van der Stigchel B, van den Bosch K, van Diggelen J, Haselager P. Intelligent decision support in medical triage: are people robust to biased advice? J Public Health (Oxf). Aug 28, 2023;45(3):689-696. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jakhar D, Kaur I. Artificial intelligence, machine learning and deep learning: definitions and differences. Clin Exp Dermatol. Jan 09, 2020;45(1):131-132. [ CrossRef ] [ Medline ]
• Ghosh M, Thirugnanam A. Introduction to artificial intelligence. In: Srinivasa KG, Siddesh GM, Sekhar SR, editors. Artificial Intelligence for Information Management: A Healthcare Perspective. Cham, Switzerland. Springer; 2021;88-44.
• Taulli T. Artificial Intelligence Basics: A Non-Technical Introduction. Berkeley, CA. Apress Berkeley; 2019.
• Patil S, Joda T, Soffe B, Awan KH, Fageeh HN, Tovani-Palone MR, et al. Efficacy of artificial intelligence in the detection of periodontal bone loss and classification of periodontal diseases: a systematic review. J Am Dent Assoc. Sep 2023;154(9):795-804.e1. [ CrossRef ] [ Medline ]
• Quek LJ, Heikkonen MR, Lau Y. Use of artificial intelligence techniques for detection of mild cognitive impairment: a systematic scoping review. J Clin Nurs. Sep 10, 2023;32(17-18):5752-5762. [ CrossRef ] [ Medline ]
• Tan D, Mohd Nasir NF, Abdul Manan H, Yahya N. Prediction of toxicity outcomes following radiotherapy using deep learning-based models: a systematic review. Cancer Radiother. Sep 2023;27(5):398-406. [ CrossRef ] [ Medline ]
• Rabilloud N, Allaume P, Acosta O, De Crevoisier R, Bourgade R, Loussouarn D, et al. Deep learning methodologies applied to digital pathology in prostate cancer: a systematic review. Diagnostics (Basel). Aug 14, 2023;13(16):2676. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Li K, Yao S, Zhang Z, Cao B, Wilson C, Kalos D, et al. Efficient gradient boosting for prognostic biomarker discovery. Bioinformatics. Mar 04, 2022;38(6):1631-1638. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chen T, Chen Y, Li H, Gao T, Tu H, Li S. Driver intent-based intersection autonomous driving collision avoidance reinforcement learning algorithm. Sensors (Basel). Dec 16, 2022;22(24):9943. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Huynh QT, Nguyen PH, Le HX, Ngo LT, Trinh NT, Tran MT, et al. Automatic acne object detection and acne severity grading using smartphone images and artificial intelligence. Diagnostics (Basel). Aug 03, 2022;12(8):1879. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brooke BS, Schwartz TA, Pawlik TM. MOOSE reporting guidelines for meta-analyses of observational studies. JAMA Surg. Aug 01, 2021;156(8):787-788. [ CrossRef ] [ Medline ]
• Scholten RJ, Clarke M, Hetherington J. The Cochrane collaboration. Eur J Clin Nutr. Aug 28, 2005;59 Suppl 1(S1):S147-S196. [ CrossRef ] [ Medline ]
• Warrens MJ. Kappa coefficients for dichotomous-nominal classifications. Adv Data Anal Classif. Apr 07, 2020;15(1):193-208. [ CrossRef ]
• Law M, Stewart D, Letts L, Pollock N, Bosch J. Guidelines for critical review of qualitative studies. McMaster University Occupational Therapy Evidence-Based Practice Research Group. URL: https://www.canchild.ca/system/tenon/assets/attachments/000/000/360/original/qualguide.pdf [accessed 2024-04-05]
• Higgins JP, Morgan RL, Rooney AA, Taylor KW, Thayer KA, Silva RA, et al. Risk of bias in non-randomized studies - of exposure (ROBINS-E). ROBINS-E tool. URL: https://www.riskofbias.info/welcome/robins-e-tool [accessed 2024-04-02]
• Nait Aicha A, Englebienne G, van Schooten KS, Pijnappels M, Kröse B. Deep learning to predict falls in older adults based on daily-life trunk accelerometry. Sensors (Basel). May 22, 2018;18(5):1654. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ribeiro NF, André J, Costa L, Santos CP. Development of a strategy to predict and detect falls using wearable sensors. J Med Syst. Apr 04, 2019;43(5):134. [ CrossRef ] [ Medline ]
• Qiu H, Rehman RZ, Yu X, Xiong S. Application of wearable inertial sensors and a new test battery for distinguishing retrospective fallers from non-fallers among community-dwelling older people. Sci Rep. Nov 05, 2018;8(1):16349. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dubois A, Bihl T, Bresciani JP. Automatic measurement of fall risk indicators in timed up and go test. Inform Health Soc Care. Sep 13, 2019;44(3):237-245. [ CrossRef ] [ Medline ]
• Noh B, Youm C, Goh E, Lee M, Park H, Jeon H, et al. XGBoost based machine learning approach to predict the risk of fall in older adults using gait outcomes. Sci Rep. Jun 09, 2021;11(1):12183. [ CrossRef ] [ Medline ]
• Hauth J, Jabri S, Kamran F, Feleke EW, Nigusie K, Ojeda LV, et al. Automated loss-of-balance event identification in older adults at risk of falls during real-world walking using wearable inertial measurement units. Sensors (Basel). Jul 07, 2021;21(14):4661. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lockhart TE, Soangra R, Yoon H, Wu T, Frames CW, Weaver R. Prediction of fall risk among community-dwelling older adults using a wearable system. Sci Rep. 2021;11(1):20976. [ CrossRef ]
• Hsu YC, Weng HH, Kuo CY, Chu TP, Tsai YH. Prediction of fall events during admission using eXtreme gradient boosting: a comparative validation study. Sci Rep. Oct 08, 2020;10(1):16777. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hu Y, Bishnoi A, Kaur R, Sowers R, Hernandez ME. Exploration of machine learning to identify community dwelling older adults with balance dysfunction using short duration accelerometer data. In: Proceedings of the 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society. 2020. Presented at: EMBC '20; July 20-24, 2020;812-815; Montreal, QC. URL: https://ieeexplore.ieee.org/document/9175871 [ CrossRef ]
• Ye C, Li J, Hao S, Liu M, Jin H, Zheng L, et al. Identification of elders at higher risk for fall with statewide electronic health records and a machine learning algorithm. Int J Med Inform. May 2020;137:104105. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Althobaiti T, Katsigiannis S, Ramzan N. Triaxial accelerometer-based falls and activities of daily life detection using machine learning. Sensors (Basel). Jul 06, 2020;20(13):3777. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tunca C, Salur G, Ersoy C. Deep learning for fall risk assessment with inertial sensors: utilizing domain knowledge in spatio-temporal gait parameters. IEEE J Biomed Health Inform. Jul 2020;24(7):1994-2005. [ CrossRef ]
• Kim K, Yun G, Park SK, Kim DH. Fall detection for the elderly based on 3-axis accelerometer and depth sensor fusion with random forest classifier. In: Proceedings of the 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2019. Presented at: EMBC '19; July 23-27, 2019;4611-4614; Berlin, Germany. URL: https://ieeexplore.ieee.org/document/8856698 [ CrossRef ]
• Gillain S, Boutaayamou M, Schwartz C, Brüls O, Bruyère O, Croisier JL, et al. Using supervised learning machine algorithm to identify future fallers based on gait patterns: a two-year longitudinal study. Exp Gerontol. Nov 2019;127:110730. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lo Y, Lynch SF, Urbanowicz RJ, Olson RS, Ritter AZ, Whitehouse CR, et al. Using machine learning on home health care assessments to predict fall risk. Stud Health Technol Inform. Aug 21, 2019;264:684-688. [ CrossRef ] [ Medline ]
• Maray N, Ngu AH, Ni J, Debnath M, Wang L. Transfer learning on small datasets for improved fall detection. Sensors (Basel). Jan 18, 2023;23(3):1105. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ladios-Martin M, Cabañero-Martínez MJ, Fernández-de-Maya J, Ballesta-López FJ, Belso-Garzas A, Zamora-Aznar FM, et al. Development of a predictive inpatient falls risk model using machine learning. J Nurs Manag. Nov 30, 2022;30(8):3777-3786. [ CrossRef ] [ Medline ]
• Eichler N, Raz S, Toledano-Shubi A, Livne D, Shimshoni I, Hel-Or H. Automatic and efficient fall risk assessment based on machine learning. Sensors (Basel). Feb 17, 2022;22(4):1557. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tang YM, Wang YH, Feng XY, Zou QS, Wang Q, Ding J, et al. Diagnostic value of a vision-based intelligent gait analyzer in screening for gait abnormalities. Gait Posture. Jan 2022;91:205-211. [ CrossRef ] [ Medline ]
• Greene BR, McManus K, Ader LG, Caulfield B. Unsupervised assessment of balance and falls risk using a smartphone and machine learning. Sensors (Basel). Jul 13, 2021;21(14):4770. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Roshdibenam V, Jogerst GJ, Butler NR, Baek S. Machine learning prediction of fall risk in older adults using timed up and go test kinematics. Sensors (Basel). May 17, 2021;21(10):3481. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dubois A, Bihl T, Bresciani JP. Identifying fall risk predictors by monitoring daily activities at home using a depth sensor coupled to machine learning algorithms. Sensors (Basel). Mar 11, 2021;21(6):1957. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vo MT, Thonglor R, Moncatar TJ, Han TD, Tejativaddhana P, Nakamura K. Fear of falling and associated factors among older adults in Southeast Asia: a systematic review. Public Health. Sep 2023;222:215-228. [ CrossRef ] [ Medline ]
• Torun E, Az A, Akdemir T, Solakoğlu GA, Açiksari K, Güngörer B. Evaluation of the risk factors for falls in the geriatric population presenting to the emergency department. Ulus Travma Acil Cerrahi Derg. Aug 2023;29(8):897-903. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Son NK, Ryu YU, Jeong HW, Jang YH, Kim HD. Comparison of 2 different exercise approaches: Tai Chi versus Otago, in community-dwelling older women. J Geriatr Phys Ther. 2016;39(2):51-57. [ CrossRef ] [ Medline ]
• Sawa R, Doi T, Tsutsumimoto K, Nakakubo S, Kurita S, Kiuchi Y, et al. Overlapping status of frailty and fear of falling: an elevated risk of incident disability in community-dwelling older adults. Aging Clin Exp Res. Sep 11, 2023;35(9):1937-1944. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Calazans JA, Permanyer I. Levels, trends, and determinants of cause-of-death diversity in a global perspective: 1990-2019. BMC Public Health. Apr 05, 2023;23(1):650. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kakara R, Bergen G, Burns E, Stevens M. Nonfatal and fatal falls among adults aged ≥65 years - United States, 2020-2021. MMWR Morb Mortal Wkly Rep. Sep 01, 2023;72(35):938-943. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dostan A, Dobson CA, Vanicek N. Relationship between stair ascent gait speed, bone density and gait characteristics of postmenopausal women. PLoS One. Mar 22, 2023;18(3):e0283333. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zheng Y, Wang X, Zhang ZK, Guo B, Dang L, He B, et al. Bushen Yijing Fang reduces fall risk in late postmenopausal women with osteopenia: a randomized double-blind and placebo-controlled trial. Sci Rep. Feb 14, 2019;9(1):2089. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Woelfle T, Bourguignon L, Lorscheider J, Kappos L, Naegelin Y, Jutzeler CR. Wearable sensor technologies to assess motor functions in people with multiple sclerosis: systematic scoping review and perspective. J Med Internet Res. Jul 27, 2023;25:e44428. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Abdollah V, Dief TN, Ralston J, Ho C, Rouhani H. Investigating the validity of a single tri-axial accelerometer mounted on the head for monitoring the activities of daily living and the timed-up and go test. Gait Posture. Oct 2021;90:137-140. [ CrossRef ] [ Medline ]
• Mielke GI, de Almeida Mendes M, Ekelund U, Rowlands AV, Reichert FF, Crochemore-Silva I. Absolute intensity thresholds for tri-axial wrist and waist accelerometer-measured movement behaviors in adults. Scand J Med Sci Sports. Sep 12, 2023;33(9):1752-1764. [ CrossRef ] [ Medline ]
• Löppönen A, Delecluse C, Suorsa K, Karavirta L, Leskinen T, Meulemans L, et al. Association of sit-to-stand capacity and free-living performance using Thigh-Worn accelerometers among 60- to 90-yr-old adults. Med Sci Sports Exerc. Sep 01, 2023;55(9):1525-1532. [ FREE Full text ] [ CrossRef ] [ Medline ]
• García-Soidán JL, Leirós-Rodríguez R, Romo-Pérez V, García-Liñeira J. Accelerometric assessment of postural balance in children: a systematic review. Diagnostics (Basel). Dec 22, 2020;11(1):8. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Leirós-Rodríguez R, García-Soidán JL, Romo-Pérez V. Analyzing the use of accelerometers as a method of early diagnosis of alterations in balance in elderly people: a systematic review. Sensors (Basel). Sep 09, 2019;19(18):3883. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 28.11.23; peer-reviewed by E Andrade, M Behzadifar, A Suárez; comments to author 09.01.24; revised version received 30.01.24; accepted 13.02.24; published 29.04.24.

©Ana González-Castro, Raquel Leirós-Rodríguez, Camino Prada-García, José Alberto Benítez-Andrades. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 29.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

• EASA Community Network

EASA publishes Artificial Intelligence Concept Paper Issue 2 ‘Guidance for Level 1 & 2 machine learning applications’

Related content, related news, get notified via email alerts.

In a significant next step on its Artificial Intelligence (AI) Roadmap , the European Union Aviation Safety Agency (EASA) has published Issue 2 of its Concept Paper on Artificial Intelligence (AI) and Machine Learning (ML) . This second issue offers the potential to enhance four aviation pillars – safety, efficiency, sustainability, and passenger experience – and positions ML at the forefront of aviation innovation. At the same time, the path to ML deployment is bringing unique challenges, particularly in safeguarding operational safety.

This issue of the EASA AI Concept Paper refines the guidance for Level 1 AI applications (those enhancing human capabilities) and deepens the exploration of 'learning assurance', 'AI explainability' and 'ethics-based assessment'. These foundation concepts are crucial for the safe and trustworthy development and implementation of AI technologies in aviation.

Going one step further, this new issue provides comprehensive guidance for the development and deployment of Level 2 AI-based systems. Level 2 AI introduces the groundbreaking concept of 'human-AI teaming' (HAT), setting the stage for AI systems that automatically take decisions under human oversight. This advancement in the authority level of AI-based systems shows the need for human guidance and design principles to ensure safe 'human-AI interaction' (HAII).

Issue 2 of the EASA AI Concept Paper marks the entry of the EASA AI Roadmap into its second phase (framework consolidation), where Rulemaking Task (RMT).0742 will facilitate the integration of the anticipated guidance from the AI Concept Paper into a comprehensive framework of generic rules and acceptable means of compliance (AMC). These rules and AMC are precisely tailored to accommodate the unique requirements of each aviation domain impacted by these new technologies.

Overall, this new AI Roadmap deliverable underscores EASA's commitment towards a future where AI and ML are integrated in aviation's successes. This vision is not just about technological advancement, but mainly about building trust in AI applications, ensuring they complement human expertise and enhance overall aviation safety and sustainability.

EASA would like to thank all the stakeholders who participated in the public consultation phase and, in so doing, contributed to the maturity of this new publication.

EASA Artificial Intelligence Concept Paper Issue 2

Easa artificial intelligence roadmap 2.0, easa artificial intelligence days — high-level conference 2024, first public deliverable for machine learning application approval (mleap) research project, easa artificial intelligence roadmap 2.0 published, easa artificial intelligence concept paper (proposed issue 2) open for consultation, five new easa research projects go live, easa publishes research agenda 2022-2024, create an account with your email or login, create an account with your social account or login.