ethics in ai research paper

An Overview of Artificial Intelligence Ethics

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Help | Advanced Search

Computer Science > Computers and Society

Title: ethics of ai: a systematic literature review of principles and challenges.

Abstract: Ethics in AI becomes a global topic of interest for both policymakers and academic researchers. In the last few years, various research organizations, lawyers, think tankers and regulatory bodies get involved in developing AI ethics guidelines and principles. However, there is still debate about the implications of these principles. We conducted a systematic literature review (SLR) study to investigate the agreement on the significance of AI principles and identify the challenging factors that could negatively impact the adoption of AI ethics principles. The results reveal that the global convergence set consists of 22 ethical principles and 15 challenges. Transparency, privacy, accountability and fairness are identified as the most common AI ethics principles. Similarly, lack of ethical knowledge and vague principles are reported as the significant challenges for considering ethics in AI. The findings of this study are the preliminary inputs for proposing a maturity model that assess the ethical capabilities of AI systems and provide best practices for further improvements.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

DBLP - CS Bibliography

Bibtex formatted citation.

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

• Table of Contents
• Random Entry
• Chronological
• Editorial Information
• About the SEP
• Editorial Board
• How to Cite the SEP
• Special Characters
• Advanced Tools
• Support the SEP
• PDFs for SEP Friends
• Make a Donation
• SEPIA for Libraries
• Entry Contents

Bibliography

Academic tools.

• Friends PDF Preview
• Author and Citation Info
• Back to Top

Ethics of Artificial Intelligence and Robotics

Artificial intelligence (AI) and robotics are digital technologies that will have significant impact on the development of humanity in the near future. They have raised fundamental questions about what we should do with these systems, what the systems themselves should do, what risks they involve, and how we can control these.

After the Introduction to the field (§1), the main themes (§2) of this article are: Ethical issues that arise with AI systems as objects , i.e., tools made and used by humans. This includes issues of privacy (§2.1) and manipulation (§2.2), opacity (§2.3) and bias (§2.4), human-robot interaction (§2.5), employment (§2.6), and the effects of autonomy (§2.7). Then AI systems as subjects , i.e., ethics for the AI systems themselves in machine ethics (§2.8) and artificial moral agency (§2.9). Finally, the problem of a possible future AI superintelligence leading to a “singularity” (§2.10). We close with a remark on the vision of AI (§3).

For each section within these themes, we provide a general explanation of the ethical issues , outline existing positions and arguments , then analyse how these play out with current technologies and finally, what policy consequences may be drawn.

1.1 Background of the Field

1.2 ai & robotics, 1.3 a note on policy, 2.1 privacy & surveillance, 2.2 manipulation of behaviour, 2.3 opacity of ai systems, 2.4 bias in decision systems, 2.5 human-robot interaction, 2.6 automation and employment, 2.7 autonomous systems, 2.8 machine ethics, 2.9 artificial moral agents, 2.10 singularity, research organizations, conferences, policy documents, other relevant pages, related entries, 1. introduction.

The ethics of AI and robotics is often focused on “concerns” of various sorts, which is a typical response to new technologies. Many such concerns turn out to be rather quaint (trains are too fast for souls); some are predictably wrong when they suggest that the technology will fundamentally change humans (telephones will destroy personal communication, writing will destroy memory, video cassettes will make going out redundant); some are broadly correct but moderately relevant (digital technology will destroy industries that make photographic film, cassette tapes, or vinyl records); but some are broadly correct and deeply relevant (cars will kill children and fundamentally change the landscape). The task of an article such as this is to analyse the issues and to deflate the non-issues.

Some technologies, like nuclear power, cars, or plastics, have caused ethical and political discussion and significant policy efforts to control the trajectory these technologies, usually only once some damage is done. In addition to such “ethical concerns”, new technologies challenge current norms and conceptual systems, which is of particular interest to philosophy. Finally, once we have understood a technology in its context, we need to shape our societal response, including regulation and law. All these features also exist in the case of new AI and Robotics technologies—plus the more fundamental fear that they may end the era of human control on Earth.

The ethics of AI and robotics has seen significant press coverage in recent years, which supports related research, but also may end up undermining it: the press often talks as if the issues under discussion were just predictions of what future technology will bring, and as though we already know what would be most ethical and how to achieve that. Press coverage thus focuses on risk, security (Brundage et al. 2018, in the Other Internet Resources section below, hereafter [OIR]), and prediction of impact (e.g., on the job market). The result is a discussion of essentially technical problems that focus on how to achieve a desired outcome. Current discussions in policy and industry are also motivated by image and public relations, where the label “ethical” is really not much more than the new “green”, perhaps used for “ethics washing”. For a problem to qualify as a problem for AI ethics would require that we do not readily know what the right thing to do is. In this sense, job loss, theft, or killing with AI is not a problem in ethics, but whether these are permissible under certain circumstances is a problem. This article focuses on the genuine problems of ethics where we do not readily know what the answers are.

A last caveat: The ethics of AI and robotics is a very young field within applied ethics, with significant dynamics, but few well-established issues and no authoritative overviews—though there is a promising outline (European Group on Ethics in Science and New Technologies 2018) and there are beginnings on societal impact (Floridi et al. 2018; Taddeo and Floridi 2018; S. Taylor et al. 2018; Walsh 2018; Bryson 2019; Gibert 2019; Whittlestone et al. 2019), and policy recommendations (AI HLEG 2019 [OIR]; IEEE 2019). So this article cannot merely reproduce what the community has achieved thus far, but must propose an ordering where little order exists.

The notion of “artificial intelligence” (AI) is understood broadly as any kind of artificial computational system that shows intelligent behaviour, i.e., complex behaviour that is conducive to reaching goals. In particular, we do not wish to restrict “intelligence” to what would require intelligence if done by humans , as Minsky had suggested (1985). This means we incorporate a range of machines, including those in “technical AI”, that show only limited abilities in learning or reasoning but excel at the automation of particular tasks, as well as machines in “general AI” that aim to create a generally intelligent agent.

AI somehow gets closer to our skin than other technologies—thus the field of “philosophy of AI”. Perhaps this is because the project of AI is to create machines that have a feature central to how we humans see ourselves, namely as feeling, thinking, intelligent beings. The main purposes of an artificially intelligent agent probably involve sensing, modelling, planning and action, but current AI applications also include perception, text analysis, natural language processing (NLP), logical reasoning, game-playing, decision support systems, data analytics, predictive analytics, as well as autonomous vehicles and other forms of robotics (P. Stone et al. 2016). AI may involve any number of computational techniques to achieve these aims, be that classical symbol-manipulating AI, inspired by natural cognition, or machine learning via neural networks (Goodfellow, Bengio, and Courville 2016; Silver et al. 2018).

Historically, it is worth noting that the term “AI” was used as above ca. 1950–1975, then came into disrepute during the “AI winter”, ca. 1975–1995, and narrowed. As a result, areas such as “machine learning”, “natural language processing” and “data science” were often not labelled as “AI”. Since ca. 2010, the use has broadened again, and at times almost all of computer science and even high-tech is lumped under “AI”. Now it is a name to be proud of, a booming industry with massive capital investment (Shoham et al. 2018), and on the edge of hype again. As Erik Brynjolfsson noted, it may allow us to

virtually eliminate global poverty, massively reduce disease and provide better education to almost everyone on the planet. (quoted in Anderson, Rainie, and Luchsinger 2018)

While AI can be entirely software, robots are physical machines that move. Robots are subject to physical impact, typically through “sensors”, and they exert physical force onto the world, typically through “actuators”, like a gripper or a turning wheel. Accordingly, autonomous cars or planes are robots, and only a minuscule portion of robots is “humanoid” (human-shaped), like in the movies. Some robots use AI, and some do not: Typical industrial robots blindly follow completely defined scripts with minimal sensory input and no learning or reasoning (around 500,000 such new industrial robots are installed each year (IFR 2019 [OIR])). It is probably fair to say that while robotics systems cause more concerns in the general public, AI systems are more likely to have a greater impact on humanity. Also, AI or robotics systems for a narrow set of tasks are less likely to cause new issues than systems that are more flexible and autonomous.

Robotics and AI can thus be seen as covering two overlapping sets of systems: systems that are only AI, systems that are only robotics, and systems that are both. We are interested in all three; the scope of this article is thus not only the intersection, but the union, of both sets.

Policy is only one of the concerns of this article. There is significant public discussion about AI ethics, and there are frequent pronouncements from politicians that the matter requires new policy, which is easier said than done: Actual technology policy is difficult to plan and enforce. It can take many forms, from incentives and funding, infrastructure, taxation, or good-will statements, to regulation by various actors, and the law. Policy for AI will possibly come into conflict with other aims of technology policy or general policy. Governments, parliaments, associations, and industry circles in industrialised countries have produced reports and white papers in recent years, and some have generated good-will slogans (“trusted/responsible/humane/human-centred/good/beneficial AI”), but is that what is needed? For a survey, see Jobin, Ienca, and Vayena (2019) and V. Müller’s list of PT-AI Policy Documents and Institutions .

For people who work in ethics and policy, there might be a tendency to overestimate the impact and threats from a new technology, and to underestimate how far current regulation can reach (e.g., for product liability). On the other hand, there is a tendency for businesses, the military, and some public administrations to “just talk” and do some “ethics washing” in order to preserve a good public image and continue as before. Actually implementing legally binding regulation would challenge existing business models and practices. Actual policy is not just an implementation of ethical theory, but subject to societal power structures—and the agents that do have the power will push against anything that restricts them. There is thus a significant risk that regulation will remain toothless in the face of economical and political power.

Though very little actual policy has been produced, there are some notable beginnings: The latest EU policy document suggests “trustworthy AI” should be lawful, ethical, and technically robust, and then spells this out as seven requirements: human oversight, technical robustness, privacy and data governance, transparency, fairness, well-being, and accountability (AI HLEG 2019 [OIR]). Much European research now runs under the slogan of “responsible research and innovation” (RRI), and “technology assessment” has been a standard field since the advent of nuclear power. Professional ethics is also a standard field in information technology, and this includes issues that are relevant in this article. Perhaps a “code of ethics” for AI engineers, analogous to the codes of ethics for medical doctors, is an option here (Véliz 2019). What data science itself should do is addressed in (L. Taylor and Purtova 2019). We also expect that much policy will eventually cover specific uses or technologies of AI and robotics, rather than the field as a whole. A useful summary of an ethical framework for AI is given in (European Group on Ethics in Science and New Technologies 2018: 13ff). On general AI policy, see Calo (2018) as well as Crawford and Calo (2016); Stahl, Timmermans, and Mittelstadt (2016); Johnson and Verdicchio (2017); and Giubilini and Savulescu (2018). A more political angle of technology is often discussed in the field of “Science and Technology Studies” (STS). As books like The Ethics of Invention (Jasanoff 2016) show, concerns in STS are often quite similar to those in ethics (Jacobs et al. 2019 [OIR]). In this article, we discuss the policy for each type of issue separately rather than for AI or robotics in general.

2. Main Debates

In this section we outline the ethical issues of human use of AI and robotics systems that can be more or less autonomous—which means we look at issues that arise with certain uses of the technologies which would not arise with others. It must be kept in mind, however, that technologies will always cause some uses to be easier, and thus more frequent, and hinder other uses. The design of technical artefacts thus has ethical relevance for their use (Houkes and Vermaas 2010; Verbeek 2011), so beyond “responsible use”, we also need “responsible design” in this field. The focus on use does not presuppose which ethical approaches are best suited for tackling these issues; they might well be virtue ethics (Vallor 2017) rather than consequentialist or value-based (Floridi et al. 2018). This section is also neutral with respect to the question whether AI systems truly have “intelligence” or other mental properties: It would apply equally well if AI and robotics are merely seen as the current face of automation (cf. Müller forthcoming-b).

There is a general discussion about privacy and surveillance in information technology (e.g., Macnish 2017; Roessler 2017), which mainly concerns the access to private data and data that is personally identifiable. Privacy has several well recognised aspects, e.g., “the right to be let alone”, information privacy, privacy as an aspect of personhood, control over information about oneself, and the right to secrecy (Bennett and Raab 2006). Privacy studies have historically focused on state surveillance by secret services but now include surveillance by other state agents, businesses, and even individuals. The technology has changed significantly in the last decades while regulation has been slow to respond (though there is the Regulation (EU) 2016/679)—the result is a certain anarchy that is exploited by the most powerful players, sometimes in plain sight, sometimes in hiding.

The digital sphere has widened greatly: All data collection and storage is now digital, our lives are increasingly digital, most digital data is connected to a single Internet, and there is more and more sensor technology in use that generates data about non-digital aspects of our lives. AI increases both the possibilities of intelligent data collection and the possibilities for data analysis. This applies to blanket surveillance of whole populations as well as to classic targeted surveillance. In addition, much of the data is traded between agents, usually for a fee.

At the same time, controlling who collects which data, and who has access, is much harder in the digital world than it was in the analogue world of paper and telephone calls. Many new AI technologies amplify the known issues. For example, face recognition in photos and videos allows identification and thus profiling and searching for individuals (Whittaker et al. 2018: 15ff). This continues using other techniques for identification, e.g., “device fingerprinting”, which are commonplace on the Internet (sometimes revealed in the “privacy policy”). The result is that “In this vast ocean of data, there is a frighteningly complete picture of us” (Smolan 2016: 1:01). The result is arguably a scandal that still has not received due public attention.

The data trail we leave behind is how our “free” services are paid for—but we are not told about that data collection and the value of this new raw material, and we are manipulated into leaving ever more such data. For the “big 5” companies (Amazon, Google/Alphabet, Microsoft, Apple, Facebook), the main data-collection part of their business appears to be based on deception, exploiting human weaknesses, furthering procrastination, generating addiction, and manipulation (Harris 2016 [OIR]). The primary focus of social media, gaming, and most of the Internet in this “surveillance economy” is to gain, maintain, and direct attention—and thus data supply. “Surveillance is the business model of the Internet” (Schneier 2015). This surveillance and attention economy is sometimes called “surveillance capitalism” (Zuboff 2019). It has caused many attempts to escape from the grasp of these corporations, e.g., in exercises of “minimalism” (Newport 2019), sometimes through the open source movement, but it appears that present-day citizens have lost the degree of autonomy needed to escape while fully continuing with their life and work. We have lost ownership of our data, if “ownership” is the right relation here. Arguably, we have lost control of our data.

These systems will often reveal facts about us that we ourselves wish to suppress or are not aware of: they know more about us than we know ourselves. Even just observing online behaviour allows insights into our mental states (Burr and Christianini 2019) and manipulation (see below section 2.2 ). This has led to calls for the protection of “derived data” (Wachter and Mittelstadt 2019). With the last sentence of his bestselling book, Homo Deus , Harari asks about the long-term consequences of AI:

What will happen to society, politics and daily life when non-conscious but highly intelligent algorithms know us better than we know ourselves? (2016: 462)

Robotic devices have not yet played a major role in this area, except for security patrolling, but this will change once they are more common outside of industry environments. Together with the “Internet of things”, the so-called “smart” systems (phone, TV, oven, lamp, virtual assistant, home,…), “smart city” (Sennett 2018), and “smart governance”, they are set to become part of the data-gathering machinery that offers more detailed data, of different types, in real time, with ever more information.

Privacy-preserving techniques that can largely conceal the identity of persons or groups are now a standard staple in data science; they include (relative) anonymisation , access control (plus encryption), and other models where computation is carried out with fully or partially encrypted input data (Stahl and Wright 2018); in the case of “differential privacy”, this is done by adding calibrated noise to encrypt the output of queries (Dwork et al. 2006; Abowd 2017). While requiring more effort and cost, such techniques can avoid many of the privacy issues. Some companies have also seen better privacy as a competitive advantage that can be leveraged and sold at a price.

One of the major practical difficulties is to actually enforce regulation, both on the level of the state and on the level of the individual who has a claim. They must identify the responsible legal entity, prove the action, perhaps prove intent, find a court that declares itself competent … and eventually get the court to actually enforce its decision. Well-established legal protection of rights such as consumer rights, product liability, and other civil liability or protection of intellectual property rights is often missing in digital products, or hard to enforce. This means that companies with a “digital” background are used to testing their products on the consumers without fear of liability while heavily defending their intellectual property rights. This “Internet Libertarianism” is sometimes taken to assume that technical solutions will take care of societal problems by themselves (Mozorov 2013).

The ethical issues of AI in surveillance go beyond the mere accumulation of data and direction of attention: They include the use of information to manipulate behaviour, online and offline, in a way that undermines autonomous rational choice. Of course, efforts to manipulate behaviour are ancient, but they may gain a new quality when they use AI systems. Given users’ intense interaction with data systems and the deep knowledge about individuals this provides, they are vulnerable to “nudges”, manipulation, and deception. With sufficient prior data, algorithms can be used to target individuals or small groups with just the kind of input that is likely to influence these particular individuals. A ’nudge‘ changes the environment such that it influences behaviour in a predictable way that is positive for the individual, but easy and cheap to avoid (Thaler & Sunstein 2008). There is a slippery slope from here to paternalism and manipulation.

Many advertisers, marketers, and online sellers will use any legal means at their disposal to maximise profit, including exploitation of behavioural biases, deception, and addiction generation (Costa and Halpern 2019 [OIR]). Such manipulation is the business model in much of the gambling and gaming industries, but it is spreading, e.g., to low-cost airlines. In interface design on web pages or in games, this manipulation uses what is called “dark patterns” (Mathur et al. 2019). At this moment, gambling and the sale of addictive substances are highly regulated, but online manipulation and addiction are not—even though manipulation of online behaviour is becoming a core business model of the Internet.

Furthermore, social media is now the prime location for political propaganda. This influence can be used to steer voting behaviour, as in the Facebook-Cambridge Analytica “scandal” (Woolley and Howard 2017; Bradshaw, Neudert, and Howard 2019) and—if successful—it may harm the autonomy of individuals (Susser, Roessler, and Nissenbaum 2019).

Improved AI “faking” technologies make what once was reliable evidence into unreliable evidence—this has already happened to digital photos, sound recordings, and video. It will soon be quite easy to create (rather than alter) “deep fake” text, photos, and video material with any desired content. Soon, sophisticated real-time interaction with persons over text, phone, or video will be faked, too. So we cannot trust digital interactions while we are at the same time increasingly dependent on such interactions.

One more specific issue is that machine learning techniques in AI rely on training with vast amounts of data. This means there will often be a trade-off between privacy and rights to data vs. technical quality of the product. This influences the consequentialist evaluation of privacy-violating practices.

The policy in this field has its ups and downs: Civil liberties and the protection of individual rights are under intense pressure from businesses’ lobbying, secret services, and other state agencies that depend on surveillance. Privacy protection has diminished massively compared to the pre-digital age when communication was based on letters, analogue telephone communications, and personal conversation and when surveillance operated under significant legal constraints.

While the EU General Data Protection Regulation (Regulation (EU) 2016/679) has strengthened privacy protection, the US and China prefer growth with less regulation (Thompson and Bremmer 2018), likely in the hope that this provides a competitive advantage. It is clear that state and business actors have increased their ability to invade privacy and manipulate people with the help of AI technology and will continue to do so to further their particular interests—unless reined in by policy in the interest of general society.

Opacity and bias are central issues in what is now sometimes called “data ethics” or “big data ethics” (Floridi and Taddeo 2016; Mittelstadt and Floridi 2016). AI systems for automated decision support and “predictive analytics” raise “significant concerns about lack of due process, accountability, community engagement, and auditing” (Whittaker et al. 2018: 18ff). They are part of a power structure in which “we are creating decision-making processes that constrain and limit opportunities for human participation” (Danaher 2016b: 245). At the same time, it will often be impossible for the affected person to know how the system came to this output, i.e., the system is “opaque” to that person. If the system involves machine learning, it will typically be opaque even to the expert, who will not know how a particular pattern was identified, or even what the pattern is. Bias in decision systems and data sets is exacerbated by this opacity. So, at least in cases where there is a desire to remove bias, the analysis of opacity and bias go hand in hand, and political response has to tackle both issues together.

Many AI systems rely on machine learning techniques in (simulated) neural networks that will extract patterns from a given dataset, with or without “correct” solutions provided; i.e., supervised, semi-supervised or unsupervised. With these techniques, the “learning” captures patterns in the data and these are labelled in a way that appears useful to the decision the system makes, while the programmer does not really know which patterns in the data the system has used. In fact, the programs are evolving, so when new data comes in, or new feedback is given (“this was correct”, “this was incorrect”), the patterns used by the learning system change. What this means is that the outcome is not transparent to the user or programmers: it is opaque. Furthermore, the quality of the program depends heavily on the quality of the data provided, following the old slogan “garbage in, garbage out”. So, if the data already involved a bias (e.g., police data about the skin colour of suspects), then the program will reproduce that bias. There are proposals for a standard description of datasets in a “datasheet” that would make the identification of such bias more feasible (Gebru et al. 2018 [OIR]). There is also significant recent literature about the limitations of machine learning systems that are essentially sophisticated data filters (Marcus 2018 [OIR]). Some have argued that the ethical problems of today are the result of technical “shortcuts” AI has taken (Cristianini forthcoming).

There are several technical activities that aim at “explainable AI”, starting with (Van Lent, Fisher, and Mancuso 1999; Lomas et al. 2012) and, more recently, a DARPA programme (Gunning 2017 [OIR]). More broadly, the demand for

a mechanism for elucidating and articulating the power structures, biases, and influences that computational artefacts exercise in society (Diakopoulos 2015: 398)

is sometimes called “algorithmic accountability reporting”. This does not mean that we expect an AI to “explain its reasoning”—doing so would require far more serious moral autonomy than we currently attribute to AI systems (see below §2.10 ).

The politician Henry Kissinger pointed out that there is a fundamental problem for democratic decision-making if we rely on a system that is supposedly superior to humans, but cannot explain its decisions. He says we may have “generated a potentially dominating technology in search of a guiding philosophy” (Kissinger 2018). Danaher (2016b) calls this problem “the threat of algocracy” (adopting the previous use of ‘algocracy’ from Aneesh 2002 [OIR], 2006). In a similar vein, Cave (2019) stresses that we need a broader societal move towards more “democratic” decision-making to avoid AI being a force that leads to a Kafka-style impenetrable suppression system in public administration and elsewhere. The political angle of this discussion has been stressed by O’Neil in her influential book Weapons of Math Destruction (2016), and by Yeung and Lodge (2019).

In the EU, some of these issues have been taken into account with the (Regulation (EU) 2016/679), which foresees that consumers, when faced with a decision based on data processing, will have a legal “right to explanation”—how far this goes and to what extent it can be enforced is disputed (Goodman and Flaxman 2017; Wachter, Mittelstadt, and Floridi 2016; Wachter, Mittelstadt, and Russell 2017). Zerilli et al. (2019) argue that there may be a double standard here, where we demand a high level of explanation for machine-based decisions despite humans sometimes not reaching that standard themselves.

Automated AI decision support systems and “predictive analytics” operate on data and produce a decision as “output”. This output may range from the relatively trivial to the highly significant: “this restaurant matches your preferences”, “the patient in this X-ray has completed bone growth”, “application to credit card declined”, “donor organ will be given to another patient”, “bail is denied”, or “target identified and engaged”. Data analysis is often used in “predictive analytics” in business, healthcare, and other fields, to foresee future developments—since prediction is easier, it will also become a cheaper commodity. One use of prediction is in “predictive policing” (NIJ 2014 [OIR]), which many fear might lead to an erosion of public liberties (Ferguson 2017) because it can take away power from the people whose behaviour is predicted. It appears, however, that many of the worries about policing depend on futuristic scenarios where law enforcement foresees and punishes planned actions, rather than waiting until a crime has been committed (like in the 2002 film “Minority Report”). One concern is that these systems might perpetuate bias that was already in the data used to set up the system, e.g., by increasing police patrols in an area and discovering more crime in that area. Actual “predictive policing” or “intelligence led policing” techniques mainly concern the question of where and when police forces will be needed most. Also, police officers can be provided with more data, offering them more control and facilitating better decisions, in workflow support software (e.g., “ArcGIS”). Whether this is problematic depends on the appropriate level of trust in the technical quality of these systems, and on the evaluation of aims of the police work itself. Perhaps a recent paper title points in the right direction here: “AI ethics in predictive policing: From models of threat to an ethics of care” (Asaro 2019).

Bias typically surfaces when unfair judgments are made because the individual making the judgment is influenced by a characteristic that is actually irrelevant to the matter at hand, typically a discriminatory preconception about members of a group. So, one form of bias is a learned cognitive feature of a person, often not made explicit. The person concerned may not be aware of having that bias—they may even be honestly and explicitly opposed to a bias they are found to have (e.g., through priming, cf. Graham and Lowery 2004). On fairness vs. bias in machine learning, see Binns (2018).

Apart from the social phenomenon of learned bias, the human cognitive system is generally prone to have various kinds of “cognitive biases”, e.g., the “confirmation bias”: humans tend to interpret information as confirming what they already believe. This second form of bias is often said to impede performance in rational judgment (Kahnemann 2011)—though at least some cognitive biases generate an evolutionary advantage, e.g., economical use of resources for intuitive judgment. There is a question whether AI systems could or should have such cognitive bias.

A third form of bias is present in data when it exhibits systematic error, e.g., “statistical bias”. Strictly, any given dataset will only be unbiased for a single kind of issue, so the mere creation of a dataset involves the danger that it may be used for a different kind of issue, and then turn out to be biased for that kind. Machine learning on the basis of such data would then not only fail to recognise the bias, but codify and automate the “historical bias”. Such historical bias was discovered in an automated recruitment screening system at Amazon (discontinued early 2017) that discriminated against women—presumably because the company had a history of discriminating against women in the hiring process. The “Correctional Offender Management Profiling for Alternative Sanctions” (COMPAS), a system to predict whether a defendant would re-offend, was found to be as successful (65.2% accuracy) as a group of random humans (Dressel and Farid 2018) and to produce more false positives and less false negatives for black defendants. The problem with such systems is thus bias plus humans placing excessive trust in the systems. The political dimensions of such automated systems in the USA are investigated in Eubanks (2018).

There are significant technical efforts to detect and remove bias from AI systems, but it is fair to say that these are in early stages: see UK Institute for Ethical AI & Machine Learning (Brownsword, Scotford, and Yeung 2017; Yeung and Lodge 2019). It appears that technological fixes have their limits in that they need a mathematical notion of fairness, which is hard to come by (Whittaker et al. 2018: 24ff; Selbst et al. 2019), as is a formal notion of “race” (see Benthall and Haynes 2019). An institutional proposal is in (Veale and Binns 2017).

Human-robot interaction (HRI) is an academic fields in its own right, which now pays significant attention to ethical matters, the dynamics of perception from both sides, and both the different interests present in and the intricacy of the social context, including co-working (e.g., Arnold and Scheutz 2017). Useful surveys for the ethics of robotics include Calo, Froomkin, and Kerr (2016); Royakkers and van Est (2016); Tzafestas (2016); a standard collection of papers is Lin, Abney, and Jenkins (2017).

While AI can be used to manipulate humans into believing and doing things (see section 2.2 ), it can also be used to drive robots that are problematic if their processes or appearance involve deception, threaten human dignity, or violate the Kantian requirement of “respect for humanity”. Humans very easily attribute mental properties to objects, and empathise with them, especially when the outer appearance of these objects is similar to that of living beings. This can be used to deceive humans (or animals) into attributing more intellectual or even emotional significance to robots or AI systems than they deserve. Some parts of humanoid robotics are problematic in this regard (e.g., Hiroshi Ishiguro’s remote-controlled Geminoids), and there are cases that have been clearly deceptive for public-relations purposes (e.g. on the abilities of Hanson Robotics’ “Sophia”). Of course, some fairly basic constraints of business ethics and law apply to robots, too: product safety and liability, or non-deception in advertisement. It appears that these existing constraints take care of many concerns that are raised. There are cases, however, where human-human interaction has aspects that appear specifically human in ways that can perhaps not be replaced by robots: care, love, and sex.

2.5.1 Example (a) Care Robots

The use of robots in health care for humans is currently at the level of concept studies in real environments, but it may become a usable technology in a few years, and has raised a number of concerns for a dystopian future of de-humanised care (A. Sharkey and N. Sharkey 2011; Robert Sparrow 2016). Current systems include robots that support human carers/caregivers (e.g., in lifting patients, or transporting material), robots that enable patients to do certain things by themselves (e.g., eat with a robotic arm), but also robots that are given to patients as company and comfort (e.g., the “Paro” robot seal). For an overview, see van Wynsberghe (2016); Nørskov (2017); Fosch-Villaronga and Albo-Canals (2019), for a survey of users Draper et al. (2014).

One reason why the issue of care has come to the fore is that people have argued that we will need robots in ageing societies. This argument makes problematic assumptions, namely that with longer lifespan people will need more care, and that it will not be possible to attract more humans to caring professions. It may also show a bias about age (Jecker forthcoming). Most importantly, it ignores the nature of automation, which is not simply about replacing humans, but about allowing humans to work more efficiently. It is not very clear that there really is an issue here since the discussion mostly focuses on the fear of robots de-humanising care, but the actual and foreseeable robots in care are assistive robots for classic automation of technical tasks. They are thus “care robots” only in a behavioural sense of performing tasks in care environments, not in the sense that a human “cares” for the patients. It appears that the success of “being cared for” relies on this intentional sense of “care”, which foreseeable robots cannot provide. If anything, the risk of robots in care is the absence of such intentional care—because less human carers may be needed. Interestingly, caring for something, even a virtual agent, can be good for the carer themselves (Lee et al. 2019). A system that pretends to care would be deceptive and thus problematic—unless the deception is countered by sufficiently large utility gain (Coeckelbergh 2016). Some robots that pretend to “care” on a basic level are available (Paro seal) and others are in the making. Perhaps feeling cared for by a machine, to some extent, is progress for come patients.

2.5.2 Example (b) Sex Robots

It has been argued by several tech optimists that humans will likely be interested in sex and companionship with robots and be comfortable with the idea (Levy 2007). Given the variation of human sexual preferences, including sex toys and sex dolls, this seems very likely: The question is whether such devices should be manufactured and promoted, and whether there should be limits in this touchy area. It seems to have moved into the mainstream of “robot philosophy” in recent times (Sullins 2012; Danaher and McArthur 2017; N. Sharkey et al. 2017 [OIR]; Bendel 2018; Devlin 2018).

Humans have long had deep emotional attachments to objects, so perhaps companionship or even love with a predictable android is attractive, especially to people who struggle with actual humans, and already prefer dogs, cats, birds, a computer or a tamagotchi . Danaher (2019b) argues against (Nyholm and Frank 2017) that these can be true friendships, and is thus a valuable goal. It certainly looks like such friendship might increase overall utility, even if lacking in depth. In these discussions there is an issue of deception, since a robot cannot (at present) mean what it says, or have feelings for a human. It is well known that humans are prone to attribute feelings and thoughts to entities that behave as if they had sentience,even to clearly inanimate objects that show no behaviour at all. Also, paying for deception seems to be an elementary part of the traditional sex industry.

Finally, there are concerns that have often accompanied matters of sex, namely consent (Frank and Nyholm 2017), aesthetic concerns, and the worry that humans may be “corrupted” by certain experiences. Old fashioned though this may seem, human behaviour is influenced by experience, and it is likely that pornography or sex robots support the perception of other humans as mere objects of desire, or even recipients of abuse, and thus ruin a deeper sexual and erotic experience. In this vein, the “Campaign Against Sex Robots” argues that these devices are a continuation of slavery and prostitution (Richardson 2016).

It seems clear that AI and robotics will lead to significant gains in productivity and thus overall wealth. The attempt to increase productivity has often been a feature of the economy, though the emphasis on “growth” is a modern phenomenon (Harari 2016: 240). However, productivity gains through automation typically mean that fewer humans are required for the same output. This does not necessarily imply a loss of overall employment, however, because available wealth increases and that can increase demand sufficiently to counteract the productivity gain. In the long run, higher productivity in industrial societies has led to more wealth overall. Major labour market disruptions have occurred in the past, e.g., farming employed over 60% of the workforce in Europe and North-America in 1800, while by 2010 it employed ca. 5% in the EU, and even less in the wealthiest countries (European Commission 2013). In the 20 years between 1950 and 1970 the number of hired agricultural workers in the UK was reduced by 50% (Zayed and Loft 2019). Some of these disruptions lead to more labour-intensive industries moving to places with lower labour cost. This is an ongoing process.

Classic automation replaced human muscle, whereas digital automation replaces human thought or information-processing—and unlike physical machines, digital automation is very cheap to duplicate (Bostrom and Yudkowsky 2014). It may thus mean a more radical change on the labour market. So, the main question is: will the effects be different this time? Will the creation of new jobs and wealth keep up with the destruction of jobs? And even if it is not different, what are the transition costs, and who bears them? Do we need to make societal adjustments for a fair distribution of costs and benefits of digital automation?

Responses to the issue of unemployment from AI have ranged from the alarmed (Frey and Osborne 2013; Westlake 2014) to the neutral (Metcalf, Keller, and Boyd 2016 [OIR]; Calo 2018; Frey 2019) to the optimistic (Brynjolfsson and McAfee 2016; Harari 2016; Danaher 2019a). In principle, the labour market effect of automation seems to be fairly well understood as involving two channels:

(i) the nature of interactions between differently skilled workers and new technologies affecting labour demand and (ii) the equilibrium effects of technological progress through consequent changes in labour supply and product markets. (Goos 2018: 362)

What currently seems to happen in the labour market as a result of AI and robotics automation is “job polarisation” or the “dumbbell” shape (Goos, Manning, and Salomons 2009): The highly skilled technical jobs are in demand and highly paid, the low skilled service jobs are in demand and badly paid, but the mid-qualification jobs in factories and offices, i.e., the majority of jobs, are under pressure and reduced because they are relatively predictable, and most likely to be automated (Baldwin 2019).

Perhaps enormous productivity gains will allow the “age of leisure” to be realised, something (Keynes 1930) had predicted to occur around 2030, assuming a growth rate of 1% per annum. Actually, we have already reached the level he anticipated for 2030, but we are still working—consuming more and inventing ever more levels of organisation. Harari explains how this economic development allowed humanity to overcome hunger, disease, and war—and now we aim for immortality and eternal bliss through AI, thus his title Homo Deus (Harari 2016: 75).

In general terms, the issue of unemployment is an issue of how goods in a society should be justly distributed. A standard view is that distributive justice should be rationally decided from behind a “veil of ignorance” (Rawls 1971), i.e., as if one does not know what position in a society one would actually be taking (labourer or industrialist, etc.). Rawls thought the chosen principles would then support basic liberties and a distribution that is of greatest benefit to the least-advantaged members of society. It would appear that the AI economy has three features that make such justice unlikely: First, it operates in a largely unregulated environment where responsibility is often hard to allocate. Second, it operates in markets that have a “winner takes all” feature where monopolies develop quickly. Third, the “new economy” of the digital service industries is based on intangible assets, also called “capitalism without capital” (Haskel and Westlake 2017). This means that it is difficult to control multinational digital corporations that do not rely on a physical plant in a particular location. These three features seem to suggest that if we leave the distribution of wealth to free market forces, the result would be a heavily unjust distribution: And this is indeed a development that we can already see.

One interesting question that has not received too much attention is whether the development of AI is environmentally sustainable: Like all computing systems, AI systems produce waste that is very hard to recycle and they consume vast amounts of energy, especially for the training of machine learning systems (and even for the “mining” of cryptocurrency). Again, it appears that some actors in this space offload such costs to the general society.

There are several notions of autonomy in the discussion of autonomous systems. A stronger notion is involved in philosophical debates where autonomy is the basis for responsibility and personhood (Christman 2003 [2018]). In this context, responsibility implies autonomy, but not inversely, so there can be systems that have degrees of technical autonomy without raising issues of responsibility. The weaker, more technical, notion of autonomy in robotics is relative and gradual: A system is said to be autonomous with respect to human control to a certain degree (Müller 2012). There is a parallel here to the issues of bias and opacity in AI since autonomy also concerns a power-relation: who is in control, and who is responsible?

Generally speaking, one question is the degree to which autonomous robots raise issues our present conceptual schemes must adapt to, or whether they just require technical adjustments. In most jurisdictions, there is a sophisticated system of civil and criminal liability to resolve such issues. Technical standards, e.g., for the safe use of machinery in medical environments, will likely need to be adjusted. There is already a field of “verifiable AI” for such safety-critical systems and for “security applications”. Bodies like the IEEE (The Institute of Electrical and Electronics Engineers) and the BSI (British Standards Institution) have produced “standards”, particularly on more technical sub-problems, such as data security and transparency. Among the many autonomous systems on land, on water, under water, in air or space, we discuss two samples: autonomous vehicles and autonomous weapons.

2.7.1 Example (a) Autonomous Vehicles

Autonomous vehicles hold the promise to reduce the very significant damage that human driving currently causes—approximately 1 million humans being killed per year, many more injured, the environment polluted, earth sealed with concrete and tarmac, cities full of parked cars, etc. However, there seem to be questions on how autonomous vehicles should behave, and how responsibility and risk should be distributed in the complicated system the vehicles operates in. (There is also significant disagreement over how long the development of fully autonomous, or “level 5” cars (SAE International 2018) will actually take.)

There is some discussion of “trolley problems” in this context. In the classic “trolley problems” (Thomson 1976; Woollard and Howard-Snyder 2016: section 2) various dilemmas are presented. The simplest version is that of a trolley train on a track that is heading towards five people and will kill them, unless the train is diverted onto a side track, but on that track there is one person, who will be killed if the train takes that side track. The example goes back to a remark in (Foot 1967: 6), who discusses a number of dilemma cases where tolerated and intended consequences of an action differ. “Trolley problems” are not supposed to describe actual ethical problems or to be solved with a “right” choice. Rather, they are thought-experiments where choice is artificially constrained to a small finite number of distinct one-off options and where the agent has perfect knowledge. These problems are used as a theoretical tool to investigate ethical intuitions and theories—especially the difference between actively doing vs. allowing something to happen, intended vs. tolerated consequences, and consequentialist vs. other normative approaches (Kamm 2016). This type of problem has reminded many of the problems encountered in actual driving and in autonomous driving (Lin 2016). It is doubtful, however, that an actual driver or autonomous car will ever have to solve trolley problems (but see Keeling 2020). While autonomous car trolley problems have received a lot of media attention (Awad et al. 2018), they do not seem to offer anything new to either ethical theory or to the programming of autonomous vehicles.

The more common ethical problems in driving, such as speeding, risky overtaking, not keeping a safe distance, etc. are classic problems of pursuing personal interest vs. the common good. The vast majority of these are covered by legal regulations on driving. Programming the car to drive “by the rules” rather than “by the interest of the passengers” or “to achieve maximum utility” is thus deflated to a standard problem of programming ethical machines (see section 2.9 ). There are probably additional discretionary rules of politeness and interesting questions on when to break the rules (Lin 2016), but again this seems to be more a case of applying standard considerations (rules vs. utility) to the case of autonomous vehicles.

Notable policy efforts in this field include the report (German Federal Ministry of Transport and Digital Infrastructure 2017), which stresses that safety is the primary objective. Rule 10 states

In the case of automated and connected driving systems, the accountability that was previously the sole preserve of the individual shifts from the motorist to the manufacturers and operators of the technological systems and to the bodies responsible for taking infrastructure, policy and legal decisions.

(See section 2.10.1 below). The resulting German and EU laws on licensing automated driving are much more restrictive than their US counterparts where “testing on consumers” is a strategy used by some companies—without informed consent of the consumers or their possible victims.

2.7.2 Example (b) Autonomous Weapons

The notion of automated weapons is fairly old:

For example, instead of fielding simple guided missiles or remotely piloted vehicles, we might launch completely autonomous land, sea, and air vehicles capable of complex, far-ranging reconnaissance and attack missions. (DARPA 1983: 1)

This proposal was ridiculed as “fantasy” at the time (Dreyfus, Dreyfus, and Athanasiou 1986: ix), but it is now a reality, at least for more easily identifiable targets (missiles, planes, ships, tanks, etc.), but not for human combatants. The main arguments against (lethal) autonomous weapon systems (AWS or LAWS), are that they support extrajudicial killings, take responsibility away from humans, and make wars or killings more likely—for a detailed list of issues see Lin, Bekey, and Abney (2008: 73–86).

It appears that lowering the hurdle to use such systems (autonomous vehicles, “fire-and-forget” missiles, or drones loaded with explosives) and reducing the probability of being held accountable would increase the probability of their use. The crucial asymmetry where one side can kill with impunity, and thus has few reasons not to do so, already exists in conventional drone wars with remote controlled weapons (e.g., US in Pakistan). It is easy to imagine a small drone that searches, identifies, and kills an individual human—or perhaps a type of human. These are the kinds of cases brought forward by the Campaign to Stop Killer Robots and other activist groups. Some seem to be equivalent to saying that autonomous weapons are indeed weapons …, and weapons kill, but we still make them in gigantic numbers. On the matter of accountability, autonomous weapons might make identification and prosecution of the responsible agents more difficult—but this is not clear, given the digital records that one can keep, at least in a conventional war. The difficulty of allocating punishment is sometimes called the “retribution gap” (Danaher 2016a).

Another question is whether using autonomous weapons in war would make wars worse, or make wars less bad. If robots reduce war crimes and crimes in war, the answer may well be positive and has been used as an argument in favour of these weapons (Arkin 2009; Müller 2016a) but also as an argument against them (Amoroso and Tamburrini 2018). Arguably the main threat is not the use of such weapons in conventional warfare, but in asymmetric conflicts or by non-state agents, including criminals.

It has also been said that autonomous weapons cannot conform to International Humanitarian Law, which requires observance of the principles of distinction (between combatants and civilians), proportionality (of force), and military necessity (of force) in military conflict (A. Sharkey 2019). It is true that the distinction between combatants and non-combatants is hard, but the distinction between civilian and military ships is easy—so all this says is that we should not construct and use such weapons if they do violate Humanitarian Law. Additional concerns have been raised that being killed by an autonomous weapon threatens human dignity, but even the defenders of a ban on these weapons seem to say that these are not good arguments:

There are other weapons, and other technologies, that also compromise human dignity. Given this, and the ambiguities inherent in the concept, it is wiser to draw on several types of objections in arguments against AWS, and not to rely exclusively on human dignity. (A. Sharkey 2019)

A lot has been made of keeping humans “in the loop” or “on the loop” in the military guidance on weapons—these ways of spelling out “meaningful control” are discussed in (Santoni de Sio and van den Hoven 2018). There have been discussions about the difficulties of allocating responsibility for the killings of an autonomous weapon, and a “responsibility gap” has been suggested (esp. Rob Sparrow 2007), meaning that neither the human nor the machine may be responsible. On the other hand, we do not assume that for every event there is someone responsible for that event, and the real issue may well be the distribution of risk (Simpson and Müller 2016). Risk analysis (Hansson 2013) indicates it is crucial to identify who is exposed to risk, who is a potential beneficiary , and who makes the decisions (Hansson 2018: 1822–1824).

Machine ethics is ethics for machines, for “ethical machines”, for machines as subjects , rather than for the human use of machines as objects. It is often not very clear whether this is supposed to cover all of AI ethics or to be a part of it (Floridi and Saunders 2004; Moor 2006; Anderson and Anderson 2011; Wallach and Asaro 2017). Sometimes it looks as though there is the (dubious) inference at play here that if machines act in ethically relevant ways, then we need a machine ethics. Accordingly, some use a broader notion:

machine ethics is concerned with ensuring that the behavior of machines toward human users, and perhaps other machines as well, is ethically acceptable. (Anderson and Anderson 2007: 15)

This might include mere matters of product safety, for example. Other authors sound rather ambitious but use a narrower notion:

AI reasoning should be able to take into account societal values, moral and ethical considerations; weigh the respective priorities of values held by different stakeholders in various multicultural contexts; explain its reasoning; and guarantee transparency. (Dignum 2018: 1, 2)

Some of the discussion in machine ethics makes the very substantial assumption that machines can, in some sense, be ethical agents responsible for their actions, or “autonomous moral agents” (see van Wynsberghe and Robbins 2019). The basic idea of machine ethics is now finding its way into actual robotics where the assumption that these machines are artificial moral agents in any substantial sense is usually not made (Winfield et al. 2019). It is sometimes observed that a robot that is programmed to follow ethical rules can very easily be modified to follow unethical rules (Vanderelst and Winfield 2018).

The idea that machine ethics might take the form of “laws” has famously been investigated by Isaac Asimov, who proposed “three laws of robotics” (Asimov 1942):

First Law—A robot may not injure a human being or, through inaction, allow a human being to come to harm. Second Law—A robot must obey the orders given it by human beings except where such orders would conflict with the First Law. Third Law—A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.

Asimov then showed in a number of stories how conflicts between these three laws will make it problematic to use them despite their hierarchical organisation.

It is not clear that there is a consistent notion of “machine ethics” since weaker versions are in danger of reducing “having an ethics” to notions that would not normally be considered sufficient (e.g., without “reflection” or even without “action”); stronger notions that move towards artificial moral agents may describe a—currently—empty set.

If one takes machine ethics to concern moral agents, in some substantial sense, then these agents can be called “artificial moral agents”, having rights and responsibilities. However, the discussion about artificial entities challenges a number of common notions in ethics and it can be very useful to understand these in abstraction from the human case (cf. Misselhorn 2020; Powers and Ganascia forthcoming).

Several authors use “artificial moral agent” in a less demanding sense, borrowing from the use of “agent” in software engineering in which case matters of responsibility and rights will not arise (Allen, Varner, and Zinser 2000). James Moor (2006) distinguishes four types of machine agents: ethical impact agents (e.g., robot jockeys), implicit ethical agents (e.g., safe autopilot), explicit ethical agents (e.g., using formal methods to estimate utility), and full ethical agents (who “can make explicit ethical judgments and generally is competent to reasonably justify them. An average adult human is a full ethical agent”.) Several ways to achieve “explicit” or “full” ethical agents have been proposed, via programming it in (operational morality), via “developing” the ethics itself (functional morality), and finally full-blown morality with full intelligence and sentience (Allen, Smit, and Wallach 2005; Moor 2006). Programmed agents are sometimes not considered “full” agents because they are “competent without comprehension”, just like the neurons in a brain (Dennett 2017; Hakli and Mäkelä 2019).

In some discussions, the notion of “moral patient” plays a role: Ethical agents have responsibilities while ethical patients have rights because harm to them matters. It seems clear that some entities are patients without being agents, e.g., simple animals that can feel pain but cannot make justified choices. On the other hand, it is normally understood that all agents will also be patients (e.g., in a Kantian framework). Usually, being a person is supposed to be what makes an entity a responsible agent, someone who can have duties and be the object of ethical concerns. Such personhood is typically a deep notion associated with phenomenal consciousness, intention and free will (Frankfurt 1971; Strawson 1998). Torrance (2011) suggests “artificial (or machine) ethics could be defined as designing machines that do things that, when done by humans, are indicative of the possession of ‘ethical status’ in those humans” (2011: 116)—which he takes to be “ethical productivity and ethical receptivity ” (2011: 117)—his expressions for moral agents and patients.

2.9.1 Responsibility for Robots

There is broad consensus that accountability, liability, and the rule of law are basic requirements that must be upheld in the face of new technologies (European Group on Ethics in Science and New Technologies 2018, 18), but the issue in the case of robots is how this can be done and how responsibility can be allocated. If the robots act, will they themselves be responsible, liable, or accountable for their actions? Or should the distribution of risk perhaps take precedence over discussions of responsibility?

Traditional distribution of responsibility already occurs: A car maker is responsible for the technical safety of the car, a driver is responsible for driving, a mechanic is responsible for proper maintenance, the public authorities are responsible for the technical conditions of the roads, etc. In general

The effects of decisions or actions based on AI are often the result of countless interactions among many actors, including designers, developers, users, software, and hardware.… With distributed agency comes distributed responsibility. (Taddeo and Floridi 2018: 751).

How this distribution might occur is not a problem that is specific to AI, but it gains particular urgency in this context (Nyholm 2018a, 2018b). In classical control engineering, distributed control is often achieved through a control hierarchy plus control loops across these hierarchies.

2.9.2 Rights for Robots

Some authors have indicated that it should be seriously considered whether current robots must be allocated rights (Gunkel 2018a, 2018b; Danaher forthcoming; Turner 2019). This position seems to rely largely on criticism of the opponents and on the empirical observation that robots and other non-persons are sometimes treated as having rights. In this vein, a “relational turn” has been proposed: If we relate to robots as though they had rights, then we might be well-advised not to search whether they “really” do have such rights (Coeckelbergh 2010, 2012, 2018). This raises the question how far such anti-realism or quasi-realism can go, and what it means then to say that “robots have rights” in a human-centred approach (Gerdes 2016). On the other side of the debate, Bryson has insisted that robots should not enjoy rights (Bryson 2010), though she considers it a possibility (Gunkel and Bryson 2014).

There is a wholly separate issue whether robots (or other AI systems) should be given the status of “legal entities” or “legal persons” in a sense natural persons, but also states, businesses, or organisations are “entities”, namely they can have legal rights and duties. The European Parliament has considered allocating such status to robots in order to deal with civil liability (EU Parliament 2016; Bertolini and Aiello 2018), but not criminal liability—which is reserved for natural persons. It would also be possible to assign only a certain subset of rights and duties to robots. It has been said that “such legislative action would be morally unnecessary and legally troublesome” because it would not serve the interest of humans (Bryson, Diamantis, and Grant 2017: 273). In environmental ethics there is a long-standing discussion about the legal rights for natural objects like trees (C. D. Stone 1972).

It has also been said that the reasons for developing robots with rights, or artificial moral patients, in the future are ethically doubtful (van Wynsberghe and Robbins 2019). In the community of “artificial consciousness” researchers there is a significant concern whether it would be ethical to create such consciousness since creating it would presumably imply ethical obligations to a sentient being, e.g., not to harm it and not to end its existence by switching it off—some authors have called for a “moratorium on synthetic phenomenology” (Bentley et al. 2018: 28f).

2.10.1 Singularity and Superintelligence

In some quarters, the aim of current AI is thought to be an “artificial general intelligence” (AGI), contrasted to a technical or “narrow” AI. AGI is usually distinguished from traditional notions of AI as a general purpose system, and from Searle’s notion of “strong AI”:

computers given the right programs can be literally said to understand and have other cognitive states. (Searle 1980: 417)

The idea of singularity is that if the trajectory of artificial intelligence reaches up to systems that have a human level of intelligence, then these systems would themselves have the ability to develop AI systems that surpass the human level of intelligence, i.e., they are “superintelligent” (see below). Such superintelligent AI systems would quickly self-improve or develop even more intelligent systems. This sharp turn of events after reaching superintelligent AI is the “singularity” from which the development of AI is out of human control and hard to predict (Kurzweil 2005: 487).

The fear that “the robots we created will take over the world” had captured human imagination even before there were computers (e.g., Butler 1863) and is the central theme in Čapek’s famous play that introduced the word “robot” (Čapek 1920). This fear was first formulated as a possible trajectory of existing AI into an “intelligence explosion” by Irvin Good:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion”, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. (Good 1965: 33)

The optimistic argument from acceleration to singularity is spelled out by Kurzweil (1999, 2005, 2012) who essentially points out that computing power has been increasing exponentially, i.e., doubling ca. every 2 years since 1970 in accordance with “Moore’s Law” on the number of transistors, and will continue to do so for some time in the future. He predicted in (Kurzweil 1999) that by 2010 supercomputers will reach human computation capacity, by 2030 “mind uploading” will be possible, and by 2045 the “singularity” will occur. Kurzweil talks about an increase in computing power that can be purchased at a given cost—but of course in recent years the funds available to AI companies have also increased enormously: Amodei and Hernandez (2018 [OIR]) thus estimate that in the years 2012–2018 the actual computing power available to train a particular AI system doubled every 3.4 months, resulting in an 300,000x increase—not the 7x increase that doubling every two years would have created.

A common version of this argument (Chalmers 2010) talks about an increase in “intelligence” of the AI system (rather than raw computing power), but the crucial point of “singularity” remains the one where further development of AI is taken over by AI systems and accelerates beyond human level. Bostrom (2014) explains in some detail what would happen at that point and what the risks for humanity are. The discussion is summarised in Eden et al. (2012); Armstrong (2014); Shanahan (2015). There are possible paths to superintelligence other than computing power increase, e.g., the complete emulation of the human brain on a computer (Kurzweil 2012; Sandberg 2013), biological paths, or networks and organisations (Bostrom 2014: 22–51).

Despite obvious weaknesses in the identification of “intelligence” with processing power, Kurzweil seems right that humans tend to underestimate the power of exponential growth. Mini-test: If you walked in steps in such a way that each step is double the previous, starting with a step of one metre, how far would you get with 30 steps? (answer: almost 3 times further than the Earth’s only permanent natural satellite.) Indeed, most progress in AI is readily attributable to the availability of processors that are faster by degrees of magnitude, larger storage, and higher investment (Müller 2018). The actual acceleration and its speeds are discussed in (Müller and Bostrom 2016; Bostrom, Dafoe, and Flynn forthcoming); Sandberg (2019) argues that progress will continue for some time.

The participants in this debate are united by being technophiles in the sense that they expect technology to develop rapidly and bring broadly welcome changes—but beyond that, they divide into those who focus on benefits (e.g., Kurzweil) and those who focus on risks (e.g., Bostrom). Both camps sympathise with “transhuman” views of survival for humankind in a different physical form, e.g., uploaded on a computer (Moravec 1990, 1998; Bostrom 2003a, 2003c). They also consider the prospects of “human enhancement” in various respects, including intelligence—often called “IA” (intelligence augmentation). It may be that future AI will be used for human enhancement, or will contribute further to the dissolution of the neatly defined human single person. Robin Hanson provides detailed speculation about what will happen economically in case human “brain emulation” enables truly intelligent robots or “ems” (Hanson 2016).

The argument from superintelligence to risk requires the assumption that superintelligence does not imply benevolence—contrary to Kantian traditions in ethics that have argued higher levels of rationality or intelligence would go along with a better understanding of what is moral and better ability to act morally (Gewirth 1978; Chalmers 2010: 36f). Arguments for risk from superintelligence say that rationality and morality are entirely independent dimensions—this is sometimes explicitly argued for as an “orthogonality thesis” (Bostrom 2012; Armstrong 2013; Bostrom 2014: 105–109).

Criticism of the singularity narrative has been raised from various angles. Kurzweil and Bostrom seem to assume that intelligence is a one-dimensional property and that the set of intelligent agents is totally-ordered in the mathematical sense—but neither discusses intelligence at any length in their books. Generally, it is fair to say that despite some efforts, the assumptions made in the powerful narrative of superintelligence and singularity have not been investigated in detail. One question is whether such a singularity will ever occur—it may be conceptually impossible, practically impossible or may just not happen because of contingent events, including people actively preventing it. Philosophically, the interesting question is whether singularity is just a “myth” (Floridi 2016; Ganascia 2017), and not on the trajectory of actual AI research. This is something that practitioners often assume (e.g., Brooks 2017 [OIR]). They may do so because they fear the public relations backlash, because they overestimate the practical problems, or because they have good reasons to think that superintelligence is an unlikely outcome of current AI research (Müller forthcoming-a). This discussion raises the question whether the concern about “singularity” is just a narrative about fictional AI based on human fears. But even if one does find negative reasons compelling and the singularity not likely to occur, there is still a significant possibility that one may turn out to be wrong. Philosophy is not on the “secure path of a science” (Kant 1791: B15), and maybe AI and robotics aren’t either (Müller 2020). So, it appears that discussing the very high-impact risk of singularity has justification even if one thinks the probability of such singularity ever occurring is very low.

2.10.2 Existential Risk from Superintelligence

Thinking about superintelligence in the long term raises the question whether superintelligence may lead to the extinction of the human species, which is called an “existential risk” (or XRisk): The superintelligent systems may well have preferences that conflict with the existence of humans on Earth, and may thus decide to end that existence—and given their superior intelligence, they will have the power to do so (or they may happen to end it because they do not really care).

Thinking in the long term is the crucial feature of this literature. Whether the singularity (or another catastrophic event) occurs in 30 or 300 or 3000 years does not really matter (Baum et al. 2019). Perhaps there is even an astronomical pattern such that an intelligent species is bound to discover AI at some point, and thus bring about its own demise. Such a “great filter” would contribute to the explanation of the “Fermi paradox” why there is no sign of life in the known universe despite the high probability of it emerging. It would be bad news if we found out that the “great filter” is ahead of us, rather than an obstacle that Earth has already passed. These issues are sometimes taken more narrowly to be about human extinction (Bostrom 2013), or more broadly as concerning any large risk for the species (Rees 2018)—of which AI is only one (Häggström 2016; Ord 2020). Bostrom also uses the category of “global catastrophic risk” for risks that are sufficiently high up the two dimensions of “scope” and “severity” (Bostrom and Ćirković 2011; Bostrom 2013).

These discussions of risk are usually not connected to the general problem of ethics under risk (e.g., Hansson 2013, 2018). The long-term view has its own methodological challenges but has produced a wide discussion: (Tegmark 2017) focuses on AI and human life “3.0” after singularity while Russell, Dewey, and Tegmark (2015) and Bostrom, Dafoe, and Flynn (forthcoming) survey longer-term policy issues in ethical AI. Several collections of papers have investigated the risks of artificial general intelligence (AGI) and the factors that might make this development more or less risk-laden (Müller 2016b; Callaghan et al. 2017; Yampolskiy 2018), including the development of non-agent AI (Drexler 2019).

2.10.3 Controlling Superintelligence?

In a narrow sense, the “control problem” is how we humans can remain in control of an AI system once it is superintelligent (Bostrom 2014: 127ff). In a wider sense, it is the problem of how we can make sure an AI system will turn out to be positive according to human perception (Russell 2019); this is sometimes called “value alignment”. How easy or hard it is to control a superintelligence depends significantly on the speed of “take-off” to a superintelligent system. This has led to particular attention to systems with self-improvement, such as AlphaZero (Silver et al. 2018).

One aspect of this problem is that we might decide a certain feature is desirable, but then find out that it has unforeseen consequences that are so negative that we would not desire that feature after all. This is the ancient problem of King Midas who wished that all he touched would turn into gold. This problem has been discussed on the occasion of various examples, such as the “paperclip maximiser” (Bostrom 2003b), or the program to optimise chess performance (Omohundro 2014).

Discussions about superintelligence include speculation about omniscient beings, the radical changes on a “latter day”, and the promise of immortality through transcendence of our current bodily form—so sometimes they have clear religious undertones (Capurro 1993; Geraci 2008, 2010; O’Connell 2017: 160ff). These issues also pose a well-known problem of epistemology: Can we know the ways of the omniscient (Danaher 2015)? The usual opponents have already shown up: A characteristic response of an atheist is

People worry that computers will get too smart and take over the world, but the real problem is that they’re too stupid and they’ve already taken over the world (Domingos 2015)

The new nihilists explain that a “techno-hypnosis” through information technologies has now become our main method of distraction from the loss of meaning (Gertz 2018). Both opponents would thus say we need an ethics for the “small” problems that occur with actual AI and robotics ( sections 2.1 through 2.9 above), and that there is less need for the “big ethics” of existential risk from AI ( section 2.10 ).

The singularity thus raises the problem of the concept of AI again. It is remarkable how imagination or “vision” has played a central role since the very beginning of the discipline at the “Dartmouth Summer Research Project” (McCarthy et al. 1955 [OIR]; Simon and Newell 1958). And the evaluation of this vision is subject to dramatic change: In a few decades, we went from the slogans “AI is impossible” (Dreyfus 1972) and “AI is just automation” (Lighthill 1973) to “AI will solve all problems” (Kurzweil 1999) and “AI may kill us all” (Bostrom 2014). This created media attention and public relations efforts, but it also raises the problem of how much of this “philosophy and ethics of AI” is really about AI rather than about an imagined technology. As we said at the outset, AI and robotics have raised fundamental questions about what we should do with these systems, what the systems themselves should do, and what risks they have in the long term. They also challenge the human view of humanity as the intelligent and dominant species on Earth. We have seen issues that have been raised and will have to watch technological and social developments closely to catch the new issues early on, develop a philosophical analysis, and learn for traditional problems of philosophy.

NOTE: Citations in the main text annotated “[OIR]” may be found in the Other Internet Resources section below, not in the Bibliography.

• Abowd, John M, 2017, “How Will Statistical Agencies Operate When All Data Are Private?”, Journal of Privacy and Confidentiality , 7(3): 1–15. doi:10.29012/jpc.v7i3.404
• AI4EU, 2019, “Outcomes from the Strategic Orientation Workshop (Deliverable 7.1)”, (June 28, 2019). https://www.ai4eu.eu/ai4eu-project-deliverables
• Allen, Colin, Iva Smit, and Wendell Wallach, 2005, “Artificial Morality: Top-down, Bottom-up, and Hybrid Approaches”, Ethics and Information Technology , 7(3): 149–155. doi:10.1007/s10676-006-0004-4
• Allen, Colin, Gary Varner, and Jason Zinser, 2000, “Prolegomena to Any Future Artificial Moral Agent”, Journal of Experimental & Theoretical Artificial Intelligence , 12(3): 251–261. doi:10.1080/09528130050111428
• Amoroso, Daniele and Guglielmo Tamburrini, 2018, “The Ethical and Legal Case Against Autonomy in Weapons Systems”, Global Jurist , 18(1): art. 20170012. doi:10.1515/gj-2017-0012
• Anderson, Janna, Lee Rainie, and Alex Luchsinger, 2018, Artificial Intelligence and the Future of Humans , Washington, DC: Pew Research Center.
• Anderson, Michael and Susan Leigh Anderson, 2007, “Machine Ethics: Creating an Ethical Intelligent Agent”, AI Magazine , 28(4): 15–26.
• ––– (eds.), 2011, Machine Ethics , Cambridge: Cambridge University Press. doi:10.1017/CBO9780511978036
• Aneesh, A., 2006, Virtual Migration: The Programming of Globalization , Durham, NC and London: Duke University Press.
• Arkin, Ronald C., 2009, Governing Lethal Behavior in Autonomous Robots , Boca Raton, FL: CRC Press.
• Armstrong, Stuart, 2013, “General Purpose Intelligence: Arguing the Orthogonality Thesis”, Analysis and Metaphysics , 12: 68–84.
• –––, 2014, Smarter Than Us , Berkeley, CA: MIRI.
• Arnold, Thomas and Matthias Scheutz, 2017, “Beyond Moral Dilemmas: Exploring the Ethical Landscape in HRI”, in Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction—HRI ’17 , Vienna, Austria: ACM Press, 445–452. doi:10.1145/2909824.3020255
• Asaro, Peter M., 2019, “AI Ethics in Predictive Policing: From Models of Threat to an Ethics of Care”, IEEE Technology and Society Magazine , 38(2): 40–53. doi:10.1109/MTS.2019.2915154
• Asimov, Isaac, 1942, “Runaround: A Short Story”, Astounding Science Fiction , March 1942. Reprinted in “I, Robot”, New York: Gnome Press 1950, 1940ff.
• Awad, Edmond, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan, 2018, “The Moral Machine Experiment”, Nature , 563(7729): 59–64. doi:10.1038/s41586-018-0637-6
• Baldwin, Richard, 2019, The Globotics Upheaval: Globalisation, Robotics and the Future of Work , New York: Oxford University Press.
• Baum, Seth D., Stuart Armstrong, Timoteus Ekenstedt, Olle Häggström, Robin Hanson, Karin Kuhlemann, Matthijs M. Maas, James D. Miller, Markus Salmela, Anders Sandberg, Kaj Sotala, Phil Torres, Alexey Turchin, and Roman V. Yampolskiy, 2019, “Long-Term Trajectories of Human Civilization”, Foresight , 21(1): 53–83. doi:10.1108/FS-04-2018-0037
• Bendel, Oliver, 2018, “Sexroboter aus Sicht der Maschinenethik”, in Handbuch Filmtheorie , Bernhard Groß and Thomas Morsch (eds.), (Springer Reference Geisteswissenschaften), Wiesbaden: Springer Fachmedien Wiesbaden, 1–19. doi:10.1007/978-3-658-17484-2_22-1
• Bennett, Colin J. and Charles Raab, 2006, The Governance of Privacy: Policy Instruments in Global Perspective , second edition, Cambridge, MA: MIT Press.
• Benthall, Sebastian and Bruce D. Haynes, 2019, “Racial Categories in Machine Learning”, in Proceedings of the Conference on Fairness, Accountability, and Transparency - FAT* ’19 , Atlanta, GA, USA: ACM Press, 289–298. doi:10.1145/3287560.3287575
• Bentley, Peter J., Miles Brundage, Olle Häggström, and Thomas Metzinger, 2018, “Should We Fear Artificial Intelligence? In-Depth Analysis”, European Parliamentary Research Service, Scientific Foresight Unit (STOA), March 2018, PE 614.547, 1–40. [ Bentley et al. 2018 available online ]
• Bertolini, Andrea and Giuseppe Aiello, 2018, “Robot Companions: A Legal and Ethical Analysis”, The Information Society , 34(3): 130–140. doi:10.1080/01972243.2018.1444249
• Binns, Reuben, 2018, “Fairness in Machine Learning: Lessons from Political Philosophy”, Proceedings of the 1st Conference on Fairness, Accountability and Transparency , in Proceedings of Machine Learning Research , 81: 149–159.
• Bostrom, Nick, 2003a, “Are We Living in a Computer Simulation?”, The Philosophical Quarterly , 53(211): 243–255. doi:10.1111/1467-9213.00309
• –––, 2003b, “Ethical Issues in Advanced Artificial Intelligence”, in Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence, Volume 2 , Iva Smit, Wendell Wallach, and G.E. Lasker (eds), (IIAS-147-2003), Tecumseh, ON: International Institute of Advanced Studies in Systems Research and Cybernetics, 12–17. [ Botstrom 2003b revised available online ]
• –––, 2003c, “Transhumanist Values”, in Ethical Issues for the Twenty-First Century , Frederick Adams (ed.), Bowling Green, OH: Philosophical Documentation Center Press.
• –––, 2012, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents”, Minds and Machines , 22(2): 71–85. doi:10.1007/s11023-012-9281-3
• –––, 2013, “Existential Risk Prevention as Global Priority”, Global Policy , 4(1): 15–31. doi:10.1111/1758-5899.12002
• –––, 2014, Superintelligence: Paths, Dangers, Strategies , Oxford: Oxford University Press.
• Bostrom, Nick and Milan M. Ćirković (eds.), 2011, Global Catastrophic Risks , New York: Oxford University Press.
• Bostrom, Nick, Allan Dafoe, and Carrick Flynn, forthcoming, “Policy Desiderata for Superintelligent AI: A Vector Field Approach (V. 4.3)”, in Ethics of Artificial Intelligence , S Matthew Liao (ed.), New York: Oxford University Press. [ Bostrom, Dafoe, and Flynn forthcoming – preprint available online ]
• Bostrom, Nick and Eliezer Yudkowsky, 2014, “The Ethics of Artificial Intelligence”, in The Cambridge Handbook of Artificial Intelligence , Keith Frankish and William M. Ramsey (eds.), Cambridge: Cambridge University Press, 316–334. doi:10.1017/CBO9781139046855.020 [ Bostrom and Yudkowsky 2014 available online ]
• Bradshaw, Samantha, Lisa-Maria Neudert, and Phil Howard, 2019, “Government Responses to Malicious Use of Social Media”, Working Paper 2019.2, Oxford: Project on Computational Propaganda. [ Bradshaw, Neudert, and Howard 2019 available online/ ]
• Brownsword, Roger, Eloise Scotford, and Karen Yeung (eds.), 2017, The Oxford Handbook of Law, Regulation and Technology , Oxford: Oxford University Press. doi:10.1093/oxfordhb/9780199680832.001.0001
• Brynjolfsson, Erik and Andrew McAfee, 2016, The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies , New York: W. W. Norton.
• Bryson, Joanna J., 2010, “Robots Should Be Slaves”, in Close Engagements with Artificial Companions: Key Social, Psychological, Ethical and Design Issues , Yorick Wilks (ed.), (Natural Language Processing 8), Amsterdam: John Benjamins Publishing Company, 63–74. doi:10.1075/nlp.8.11bry
• –––, 2019, “The Past Decade and Future of Ai’s Impact on Society”, in Towards a New Enlightenment: A Transcendent Decade , Madrid: Turner - BVVA. [ Bryson 2019 available online ]
• Bryson, Joanna J., Mihailis E. Diamantis, and Thomas D. Grant, 2017, “Of, for, and by the People: The Legal Lacuna of Synthetic Persons”, Artificial Intelligence and Law , 25(3): 273–291. doi:10.1007/s10506-017-9214-9
• Burr, Christopher and Nello Cristianini, 2019, “Can Machines Read Our Minds?”, Minds and Machines , 29(3): 461–494. doi:10.1007/s11023-019-09497-4
• Butler, Samuel, 1863, “Darwin among the Machines: Letter to the Editor”, Letter in The Press (Christchurch) , 13 June 1863. [ Butler 1863 available online ]
• Callaghan, Victor, James Miller, Roman Yampolskiy, and Stuart Armstrong (eds.), 2017, The Technological Singularity: Managing the Journey , (The Frontiers Collection), Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978-3-662-54033-6
• Calo, Ryan, 2018, “Artificial Intelligence Policy: A Primer and Roadmap”, University of Bologna Law Review , 3(2): 180-218. doi:10.6092/ISSN.2531-6133/8670
• Calo, Ryan, A. Michael Froomkin, and Ian Kerr (eds.), 2016, Robot Law , Cheltenham: Edward Elgar.
• Čapek, Karel, 1920, R.U.R. , Prague: Aventium. Translated by Peter Majer and Cathy Porter, London: Methuen, 1999.
• Capurro, Raphael, 1993, “Ein Grinsen Ohne Katze: Von der Vergleichbarkeit Zwischen ‘Künstlicher Intelligenz’ und ‘Getrennten Intelligenzen’”, Zeitschrift für philosophische Forschung , 47: 93–102.
• Cave, Stephen, 2019, “To Save Us from a Kafkaesque Future, We Must Democratise AI”, The Guardian , 04 January 2019. [ Cave 2019 available online ]
• Chalmers, David J., 2010, “The Singularity: A Philosophical Analysis”, Journal of Consciousness Studies , 17(9–10): 7–65. [ Chalmers 2010 available online ]
• Christman, John, 2003 [2018], “Autonomy in Moral and Political Philosophy”, (Spring 2018) Stanford Encyclopedia of Philosophy (EDITION NEEDED), URL = < https://plato.stanford.edu/archives/spr2018/entries/autonomy-moral/ >
• Coeckelbergh, Mark, 2010, “Robot Rights? Towards a Social-Relational Justification of Moral Consideration”, Ethics and Information Technology , 12(3): 209–221. doi:10.1007/s10676-010-9235-5
• –––, 2012, Growing Moral Relations: Critique of Moral Status Ascription , London: Palgrave. doi:10.1057/9781137025968
• –––, 2016, “Care Robots and the Future of ICT-Mediated Elderly Care: A Response to Doom Scenarios”, AI & Society , 31(4): 455–462. doi:10.1007/s00146-015-0626-3
• –––, 2018, “What Do We Mean by a Relational Ethics? Growing a Relational Approach to the Moral Standing of Plants, Robots and Other Non-Humans”, in Plant Ethics: Concepts and Applications , Angela Kallhoff, Marcello Di Paola, and Maria Schörgenhumer (eds.), London: Routledge, 110–121.
• Crawford, Kate and Ryan Calo, 2016, “There Is a Blind Spot in AI Research”, Nature , 538(7625): 311–313. doi:10.1038/538311a
• Cristianini, Nello, forthcoming, “Shortcuts to Artificial Intelligence”, in Machines We Trust , Marcello Pelillo and Teresa Scantamburlo (eds.), Cambridge, MA: MIT Press. [ Cristianini forthcoming – preprint available online ]
• Danaher, John, 2015, “Why AI Doomsayers Are Like Sceptical Theists and Why It Matters”, Minds and Machines , 25(3): 231–246. doi:10.1007/s11023-015-9365-y
• –––, 2016a, “Robots, Law and the Retribution Gap”, Ethics and Information Technology , 18(4): 299–309. doi:10.1007/s10676-016-9403-3
• –––, 2016b, “The Threat of Algocracy: Reality, Resistance and Accommodation”, Philosophy & Technology , 29(3): 245–268. doi:10.1007/s13347-015-0211-1
• –––, 2019a, Automation and Utopia: Human Flourishing in a World without Work , Cambridge, MA: Harvard University Press.
• –––, 2019b, “The Philosophical Case for Robot Friendship”, Journal of Posthuman Studies , 3(1): 5–24. doi:10.5325/jpoststud.3.1.0005
• –––, forthcoming, “Welcoming Robots into the Moral Circle: A Defence of Ethical Behaviourism”, Science and Engineering Ethics , first online: 20 June 2019. doi:10.1007/s11948-019-00119-x
• Danaher, John and Neil McArthur (eds.), 2017, Robot Sex: Social and Ethical Implications , Boston, MA: MIT Press.
• DARPA, 1983, “Strategic Computing. New-Generation Computing Technology: A Strategic Plan for Its Development an Application to Critical Problems in Defense”, ADA141982, 28 October 1983. [ DARPA 1983 available online ]
• Dennett, Daniel C, 2017, From Bacteria to Bach and Back: The Evolution of Minds , New York: W.W. Norton.
• Devlin, Kate, 2018, Turned On: Science, Sex and Robots , London: Bloomsbury.
• Diakopoulos, Nicholas, 2015, “Algorithmic Accountability: Journalistic Investigation of Computational Power Structures”, Digital Journalism , 3(3): 398–415. doi:10.1080/21670811.2014.976411
• Dignum, Virginia, 2018, “Ethics in Artificial Intelligence: Introduction to the Special Issue”, Ethics and Information Technology , 20(1): 1–3. doi:10.1007/s10676-018-9450-z
• Domingos, Pedro, 2015, The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World , London: Allen Lane.
• Draper, Heather, Tom Sorell, Sandra Bedaf, Dag Sverre Syrdal, Carolina Gutierrez-Ruiz, Alexandre Duclos, and Farshid Amirabdollahian, 2014, “Ethical Dimensions of Human-Robot Interactions in the Care of Older People: Insights from 21 Focus Groups Convened in the UK, France and the Netherlands”, in International Conference on Social Robotics 2014 , Michael Beetz, Benjamin Johnston, and Mary-Anne Williams (eds.), (Lecture Notes in Artificial Intelligence 8755), Cham: Springer International Publishing, 135–145. doi:10.1007/978-3-319-11973-1_14
• Dressel, Julia and Hany Farid, 2018, “The Accuracy, Fairness, and Limits of Predicting Recidivism”, Science Advances , 4(1): eaao5580. doi:10.1126/sciadv.aao5580
• Drexler, K. Eric, 2019, “Reframing Superintelligence: Comprehensive AI Services as General Intelligence”, FHI Technical Report, 2019-1, 1-210. [ Drexler 2019 available online ]
• Dreyfus, Hubert L., 1972, What Computers Still Can’t Do: A Critique of Artificial Reason , second edition, Cambridge, MA: MIT Press 1992.
• Dreyfus, Hubert L., Stuart E. Dreyfus, and Tom Athanasiou, 1986, Mind over Machine: The Power of Human Intuition and Expertise in the Era of the Computer , New York: Free Press.
• Dwork, Cynthia, Frank McSherry, Kobbi Nissim, and Adam Smith, 2006, Calibrating Noise to Sensitivity in Private Data Analysis , Berlin, Heidelberg.
• Eden, Amnon H., James H. Moor, Johnny H. Søraker, and Eric Steinhart (eds.), 2012, Singularity Hypotheses: A Scientific and Philosophical Assessment , (The Frontiers Collection), Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978-3-642-32560-1
• Eubanks, Virginia, 2018, Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor , London: St. Martin’s Press.
• European Commission, 2013, “How Many People Work in Agriculture in the European Union? An Answer Based on Eurostat Data Sources”, EU Agricultural Economics Briefs , 8 (July 2013). [ Anonymous 2013 available online ]
• European Group on Ethics in Science and New Technologies, 2018, “Statement on Artificial Intelligence, Robotics and ‘Autonomous’ Systems”, 9 March 2018, European Commission, Directorate-General for Research and Innovation, Unit RTD.01. [ European Group 2018 available online ]
• Ferguson, Andrew Guthrie, 2017, The Rise of Big Data Policing: Surveillance, Race, and the Future of Law Enforcement , New York: NYU Press.
• Floridi, Luciano, 2016, “Should We Be Afraid of AI? Machines Seem to Be Getting Smarter and Smarter and Much Better at Human Jobs, yet True AI Is Utterly Implausible. Why?”, Aeon , 9 May 2016. URL = < Floridi 2016 available online >
• Floridi, Luciano, Josh Cowls, Monica Beltrametti, Raja Chatila, Patrice Chazerand, Virginia Dignum, Christoph Luetge, Robert Madelin, Ugo Pagallo, Francesca Rossi, Burkhard Schafer, Peggy Valcke, and Effy Vayena, 2018, “AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations”, Minds and Machines , 28(4): 689–707. doi:10.1007/s11023-018-9482-5
• Floridi, Luciano and Jeff W. Sanders, 2004, “On the Morality of Artificial Agents”, Minds and Machines , 14(3): 349–379. doi:10.1023/B:MIND.0000035461.63578.9d
• Floridi, Luciano and Mariarosaria Taddeo, 2016, “What Is Data Ethics?”, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 374(2083): 20160360. doi:10.1098/rsta.2016.0360
• Foot, Philippa, 1967, “The Problem of Abortion and the Doctrine of the Double Effect”, Oxford Review , 5: 5–15.
• Fosch-Villaronga, Eduard and Jordi Albo-Canals, 2019, “‘I’ll Take Care of You,’ Said the Robot”, Paladyn, Journal of Behavioral Robotics , 10(1): 77–93. doi:10.1515/pjbr-2019-0006
• Frank, Lily and Sven Nyholm, 2017, “Robot Sex and Consent: Is Consent to Sex between a Robot and a Human Conceivable, Possible, and Desirable?”, Artificial Intelligence and Law , 25(3): 305–323. doi:10.1007/s10506-017-9212-y
• Frankfurt, Harry G., 1971, “Freedom of the Will and the Concept of a Person”, The Journal of Philosophy , 68(1): 5–20.
• Frey, Carl Benedict, 2019, The Technology Trap: Capital, Labour, and Power in the Age of Automation , Princeton, NJ: Princeton University Press.
• Frey, Carl Benedikt and Michael A. Osborne, 2013, “The Future of Employment: How Susceptible Are Jobs to Computerisation?”, Oxford Martin School Working Papers, 17 September 2013. [ Frey and Osborne 2013 available online ]
• Ganascia, Jean-Gabriel, 2017, Le Mythe De La Singularité , Paris: Éditions du Seuil.
• EU Parliament, 2016, “Draft Report with Recommendations to the Commission on Civil Law Rules on Robotics (2015/2103(Inl))”, Committee on Legal Affairs , 10.11.2016. https://www.europarl.europa.eu/doceo/document/A-8-2017-0005_EN.html
• EU Regulation, 2016/679, “General Data Protection Regulation: Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/46/Ec”, Official Journal of the European Union , 119 (4 May 2016), 1–88. [ Regulation (EU) 2016/679 available online ]
• Geraci, Robert M., 2008, “Apocalyptic AI: Religion and the Promise of Artificial Intelligence”, Journal of the American Academy of Religion , 76(1): 138–166. doi:10.1093/jaarel/lfm101
• –––, 2010, Apocalyptic AI: Visions of Heaven in Robotics, Artificial Intelligence, and Virtual Reality , Oxford: Oxford University Press. doi:10.1093/acprof:oso/9780195393026.001.0001
• Gerdes, Anne, 2016, “The Issue of Moral Consideration in Robot Ethics”, ACM SIGCAS Computers and Society , 45(3): 274–279. doi:10.1145/2874239.2874278
• German Federal Ministry of Transport and Digital Infrastructure, 2017, “Report of the Ethics Commission: Automated and Connected Driving”, June 2017, 1–36. [ GFMTDI 2017 available online ]
• Gertz, Nolen, 2018, Nihilism and Technology , London: Rowman & Littlefield.
• Gewirth, Alan, 1978, “The Golden Rule Rationalized”, Midwest Studies in Philosophy , 3(1): 133–147. doi:10.1111/j.1475-4975.1978.tb00353.x
• Gibert, Martin, 2019, “Éthique Artificielle (Version Grand Public)”, in L’Encyclopédie Philosophique , Maxime Kristanek (ed.), accessed: 16 April 2020, URL = < Gibert 2019 available online >
• Giubilini, Alberto and Julian Savulescu, 2018, “The Artificial Moral Advisor. The ‘Ideal Observer’ Meets Artificial Intelligence”, Philosophy & Technology , 31(2): 169–188. doi:10.1007/s13347-017-0285-z
• Good, Irving John, 1965, “Speculations Concerning the First Ultraintelligent Machine”, in Advances in Computers 6 , Franz L. Alt and Morris Rubinoff (eds.), New York & London: Academic Press, 31–88. doi:10.1016/S0065-2458(08)60418-0
• Goodfellow, Ian, Yoshua Bengio, and Aaron Courville, 2016, Deep Learning , Cambridge, MA: MIT Press.
• Goodman, Bryce and Seth Flaxman, 2017, “European Union Regulations on Algorithmic Decision-Making and a ‘Right to Explanation’”, AI Magazine , 38(3): 50–57. doi:10.1609/aimag.v38i3.2741
• Goos, Maarten, 2018, “The Impact of Technological Progress on Labour Markets: Policy Challenges”, Oxford Review of Economic Policy , 34(3): 362–375. doi:10.1093/oxrep/gry002
• Goos, Maarten, Alan Manning, and Anna Salomons, 2009, “Job Polarization in Europe”, American Economic Review , 99(2): 58–63. doi:10.1257/aer.99.2.58
• Graham, Sandra and Brian S. Lowery, 2004, “Priming Unconscious Racial Stereotypes about Adolescent Offenders”, Law and Human Behavior , 28(5): 483–504. doi:10.1023/B:LAHU.0000046430.65485.1f
• Gunkel, David J., 2018a, “The Other Question: Can and Should Robots Have Rights?”, Ethics and Information Technology , 20(2): 87–99. doi:10.1007/s10676-017-9442-4
• –––, 2018b, Robot Rights , Boston, MA: MIT Press.
• Gunkel, David J. and Joanna J. Bryson (eds.), 2014, Machine Morality: The Machine as Moral Agent and Patient special issue of Philosophy & Technology , 27(1): 1–142.
• Häggström, Olle, 2016, Here Be Dragons: Science, Technology and the Future of Humanity , Oxford: Oxford University Press. doi:10.1093/acprof:oso/9780198723547.001.0001
• Hakli, Raul and Pekka Mäkelä, 2019, “Moral Responsibility of Robots and Hybrid Agents”, The Monist , 102(2): 259–275. doi:10.1093/monist/onz009
• Hanson, Robin, 2016, The Age of Em: Work, Love and Life When Robots Rule the Earth , Oxford: Oxford University Press.
• Hansson, Sven Ove, 2013, The Ethics of Risk: Ethical Analysis in an Uncertain World , New York: Palgrave Macmillan.
• –––, 2018, “How to Perform an Ethical Risk Analysis (eRA)”, Risk Analysis , 38(9): 1820–1829. doi:10.1111/risa.12978
• Harari, Yuval Noah, 2016, Homo Deus: A Brief History of Tomorrow , New York: Harper.
• Haskel, Jonathan and Stian Westlake, 2017, Capitalism without Capital: The Rise of the Intangible Economy , Princeton, NJ: Princeton University Press.
• Houkes, Wybo and Pieter E. Vermaas, 2010, Technical Functions: On the Use and Design of Artefacts , (Philosophy of Engineering and Technology 1), Dordrecht: Springer Netherlands. doi:10.1007/978-90-481-3900-2
• IEEE, 2019, Ethically Aligned Design: A Vision for Prioritizing Human Well-Being with Autonomous and Intelligent Systems (First Version), < IEEE 2019 available online >.
• Jasanoff, Sheila, 2016, The Ethics of Invention: Technology and the Human Future , New York: Norton.
• Jecker, Nancy S., forthcoming, Ending Midlife Bias: New Values for Old Age , New York: Oxford University Press.
• Jobin, Anna, Marcello Ienca, and Effy Vayena, 2019, “The Global Landscape of AI Ethics Guidelines”, Nature Machine Intelligence , 1(9): 389–399. doi:10.1038/s42256-019-0088-2
• Johnson, Deborah G. and Mario Verdicchio, 2017, “Reframing AI Discourse”, Minds and Machines , 27(4): 575–590. doi:10.1007/s11023-017-9417-6
• Kahnemann, Daniel, 2011, Thinking Fast and Slow , London: Macmillan.
• Kamm, Frances Myrna, 2016, The Trolley Problem Mysteries , Eric Rakowski (ed.), Oxford: Oxford University Press. doi:10.1093/acprof:oso/9780190247157.001.0001
• Kant, Immanuel, 1781/1787, Kritik der reinen Vernunft . Translated as Critique of Pure Reason , Norman Kemp Smith (trans.), London: Palgrave Macmillan, 1929.
• Keeling, Geoff, 2020, “Why Trolley Problems Matter for the Ethics of Automated Vehicles”, Science and Engineering Ethics , 26(1): 293–307. doi:10.1007/s11948-019-00096-1
• Keynes, John Maynard, 1930, “Economic Possibilities for Our Grandchildren”. Reprinted in his Essays in Persuasion , New York: Harcourt Brace, 1932, 358–373.
• Kissinger, Henry A., 2018, “How the Enlightenment Ends: Philosophically, Intellectually—in Every Way—Human Society Is Unprepared for the Rise of Artificial Intelligence”, The Atlantic , June 2018. [ Kissinger 2018 available online ]
• Kurzweil, Ray, 1999, The Age of Spiritual Machines: When Computers Exceed Human Intelligence , London: Penguin.
• –––, 2005, The Singularity Is Near: When Humans Transcend Biology , London: Viking.
• –––, 2012, How to Create a Mind: The Secret of Human Thought Revealed , New York: Viking.
• Lee, Minha, Sander Ackermans, Nena van As, Hanwen Chang, Enzo Lucas, and Wijnand IJsselsteijn, 2019, “Caring for Vincent: A Chatbot for Self-Compassion”, in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems—CHI ’19 , Glasgow, Scotland: ACM Press, 1–13. doi:10.1145/3290605.3300932
• Levy, David, 2007, Love and Sex with Robots: The Evolution of Human-Robot Relationships , New York: Harper & Co.
• Lighthill, James, 1973, “Artificial Intelligence: A General Survey”, Artificial intelligence: A Paper Symposion , London: Science Research Council. [ Lighthill 1973 available online ]
• Lin, Patrick, 2016, “Why Ethics Matters for Autonomous Cars”, in Autonomous Driving , Markus Maurer, J. Christian Gerdes, Barbara Lenz, and Hermann Winner (eds.), Berlin, Heidelberg: Springer Berlin Heidelberg, 69–85. doi:10.1007/978-3-662-48847-8_4
• Lin, Patrick, Keith Abney, and Ryan Jenkins (eds.), 2017, Robot Ethics 2.0: From Autonomous Cars to Artificial Intelligence , New York: Oxford University Press. doi:10.1093/oso/9780190652951.001.0001
• Lin, Patrick, George Bekey, and Keith Abney, 2008, “Autonomous Military Robotics: Risk, Ethics, and Design”, ONR report, California Polytechnic State University, San Luis Obispo, 20 December 2008), 112 pp. [ Lin, Bekey, and Abney 2008 available online ]
• Lomas, Meghann, Robert Chevalier, Ernest Vincent Cross, Robert Christopher Garrett, John Hoare, and Michael Kopack, 2012, “Explaining Robot Actions”, in Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction—HRI ’12 , Boston, MA: ACM Press, 187–188. doi:10.1145/2157689.2157748
• Macnish, Kevin, 2017, The Ethics of Surveillance: An Introduction , London: Routledge.
• Mathur, Arunesh, Gunes Acar, Michael J. Friedman, Elena Lucherini, Jonathan Mayer, Marshini Chetty, and Arvind Narayanan, 2019, “Dark Patterns at Scale: Findings from a Crawl of 11K Shopping Websites”, Proceedings of the ACM on Human-Computer Interaction , 3(CSCW): art. 81. doi:10.1145/3359183
• Minsky, Marvin, 1985, The Society of Mind , New York: Simon & Schuster.
• Misselhorn, Catrin, 2020, “Artificial Systems with Moral Capacities? A Research Design and Its Implementation in a Geriatric Care System”, Artificial Intelligence , 278: art. 103179. doi:10.1016/j.artint.2019.103179
• Mittelstadt, Brent Daniel and Luciano Floridi, 2016, “The Ethics of Big Data: Current and Foreseeable Issues in Biomedical Contexts”, Science and Engineering Ethics , 22(2): 303–341. doi:10.1007/s11948-015-9652-2
• Moor, James H., 2006, “The Nature, Importance, and Difficulty of Machine Ethics”, IEEE Intelligent Systems , 21(4): 18–21. doi:10.1109/MIS.2006.80
• Moravec, Hans, 1990, Mind Children , Cambridge, MA: Harvard University Press.
• –––, 1998, Robot: Mere Machine to Transcendent Mind , New York: Oxford University Press.
• Mozorov, Eygeny, 2013, To Save Everything, Click Here: The Folly of Technological Solutionism , New York: Public Affairs.
• Müller, Vincent C., 2012, “Autonomous Cognitive Systems in Real-World Environments: Less Control, More Flexibility and Better Interaction”, Cognitive Computation , 4(3): 212–215. doi:10.1007/s12559-012-9129-4
• –––, 2016a, “Autonomous Killer Robots Are Probably Good News”, In Drones and Responsibility: Legal, Philosophical and Socio-Technical Perspectives on the Use of Remotely Controlled Weapons , Ezio Di Nucci and Filippo Santoni de Sio (eds.), London: Ashgate, 67–81.
• ––– (ed.), 2016b, Risks of Artificial Intelligence , London: Chapman & Hall - CRC Press. doi:10.1201/b19187
• –––, 2018, “In 30 Schritten zum Mond? Zukünftiger Fortschritt in der KI”, Medienkorrespondenz , 20: 5–15. [ Müller 2018 available online ]
• –––, 2020, “Measuring Progress in Robotics: Benchmarking and the ‘Measure-Target Confusion’”, in Metrics of Sensory Motor Coordination and Integration in Robots and Animals , Fabio Bonsignorio, Elena Messina, Angel P. del Pobil, and John Hallam (eds.), (Cognitive Systems Monographs 36), Cham: Springer International Publishing, 169–179. doi:10.1007/978-3-030-14126-4_9
• –––, forthcoming-a, Can Machines Think? Fundamental Problems of Artificial Intelligence , New York: Oxford University Press.
• ––– (ed.), forthcoming-b, Oxford Handbook of the Philosophy of Artificial Intelligence , New York: Oxford University Press.
• Müller, Vincent C. and Nick Bostrom, 2016, “Future Progress in Artificial Intelligence: A Survey of Expert Opinion”, in Fundamental Issues of Artificial Intelligence , Vincent C. Müller (ed.), Cham: Springer International Publishing, 555–572. doi:10.1007/978-3-319-26485-1_33
• Newport, Cal, 2019, Digital Minimalism: On Living Better with Less Technology , London: Penguin.
• Nørskov, Marco (ed.), 2017, Social Robots , London: Routledge.
• Nyholm, Sven, 2018a, “Attributing Agency to Automated Systems: Reflections on Human–Robot Collaborations and Responsibility-Loci”, Science and Engineering Ethics , 24(4): 1201–1219. doi:10.1007/s11948-017-9943-x
• –––, 2018b, “The Ethics of Crashes with Self-Driving Cars: A Roadmap, II”, Philosophy Compass , 13(7): e12506. doi:10.1111/phc3.12506
• Nyholm, Sven, and Lily Frank, 2017, “From Sex Robots to Love Robots: Is Mutual Love with a Robot Possible?”, in Danaher and McArthur 2017: 219–243.
• O’Connell, Mark, 2017, To Be a Machine: Adventures among Cyborgs, Utopians, Hackers, and the Futurists Solving the Modest Problem of Death , London: Granta.
• O’Neil, Cathy, 2016, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy , Largo, ML: Crown.
• Omohundro, Steve, 2014, “Autonomous Technology and the Greater Human Good”, Journal of Experimental & Theoretical Artificial Intelligence , 26(3): 303–315. doi:10.1080/0952813X.2014.895111
• Ord, Toby, 2020, The Precipice: Existential Risk and the Future of Humanity , London: Bloomsbury.
• Powers, Thomas M. and Jean-Gabriel Ganascia, forthcoming, “The Ethics of the Ethics of AI”, in Oxford Handbook of Ethics of Artificial Intelligence , Markus D. Dubber, Frank Pasquale, and Sunnit Das (eds.), New York: Oxford.
• Rawls, John, 1971, A Theory of Justice , Cambridge, MA: Belknap Press.
• Rees, Martin, 2018, On the Future: Prospects for Humanity , Princeton: Princeton University Press.
• Richardson, Kathleen, 2016, “Sex Robot Matters: Slavery, the Prostituted, and the Rights of Machines”, IEEE Technology and Society Magazine , 35(2): 46–53. doi:10.1109/MTS.2016.2554421
• Roessler, Beate, 2017, “Privacy as a Human Right”, Proceedings of the Aristotelian Society , 117(2): 187–206. doi:10.1093/arisoc/aox008
• Royakkers, Lambèr and Rinie van Est, 2016, Just Ordinary Robots: Automation from Love to War , Boca Raton, LA: CRC Press, Taylor & Francis. doi:10.1201/b18899
• Russell, Stuart, 2019, Human Compatible: Artificial Intelligence and the Problem of Control , New York: Viking.
• Russell, Stuart, Daniel Dewey, and Max Tegmark, 2015, “Research Priorities for Robust and Beneficial Artificial Intelligence”, AI Magazine , 36(4): 105–114. doi:10.1609/aimag.v36i4.2577
• SAE International, 2018, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for on-Road Motor Vehicles”, J3016_201806, 15 June 2018. [ SAE International 2015 available online ]
• Sandberg, Anders, 2013, “Feasibility of Whole Brain Emulation”, in Philosophy and Theory of Artificial Intelligence , Vincent C. Müller (ed.), (Studies in Applied Philosophy, Epistemology and Rational Ethics, 5), Berlin, Heidelberg: Springer Berlin Heidelberg, 251–264. doi:10.1007/978-3-642-31674-6_19
• –––, 2019, “There Is Plenty of Time at the Bottom: The Economics, Risk and Ethics of Time Compression”, Foresight , 21(1): 84–99. doi:10.1108/FS-04-2018-0044
• Santoni de Sio, Filippo and Jeroen van den Hoven, 2018, “Meaningful Human Control over Autonomous Systems: A Philosophical Account”, Frontiers in Robotics and AI , 5(February): 15. doi:10.3389/frobt.2018.00015
• Schneier, Bruce, 2015, Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World , New York: W. W. Norton.
• Searle, John R., 1980, “Minds, Brains, and Programs”, Behavioral and Brain Sciences , 3(3): 417–424. doi:10.1017/S0140525X00005756
• Selbst, Andrew D., Danah Boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi, 2019, “Fairness and Abstraction in Sociotechnical Systems”, in Proceedings of the Conference on Fairness, Accountability, and Transparency—FAT* ’19 , Atlanta, GA: ACM Press, 59–68. doi:10.1145/3287560.3287598
• Sennett, Richard, 2018, Building and Dwelling: Ethics for the City , London: Allen Lane.
• Shanahan, Murray, 2015, The Technological Singularity , Cambridge, MA: MIT Press.
• Sharkey, Amanda, 2019, “Autonomous Weapons Systems, Killer Robots and Human Dignity”, Ethics and Information Technology , 21(2): 75–87. doi:10.1007/s10676-018-9494-0
• Sharkey, Amanda and Noel Sharkey, 2011, “The Rights and Wrongs of Robot Care”, in Robot Ethics: The Ethical and Social Implications of Robotics , Patrick Lin, Keith Abney and George Bekey (eds.), Cambridge, MA: MIT Press, 267–282.
• Shoham, Yoav, Perrault Raymond, Brynjolfsson Erik, Jack Clark, James Manyika, Juan Carlos Niebles, … Zoe Bauer, 2018, “The AI Index 2018 Annual Report”, 17 December 2018, Stanford, CA: AI Index Steering Committee, Human-Centered AI Initiative, Stanford University. [ Shoam et al. 2018 available online ]
• SIENNA, 2019, “Deliverable Report D4.4: Ethical Issues in Artificial Intelligence and Robotics”, June 2019, published by the SIENNA project (Stakeholder-informed ethics for new technologies with high socio-economic and human rights impact), University of Twente, pp. 1–103. [ SIENNA 2019 available online ]
• Silver, David, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis, 2018, “A General Reinforcement Learning Algorithm That Masters Chess, Shogi, and Go through Self-Play”, Science , 362(6419): 1140–1144. doi:10.1126/science.aar6404
• Simon, Herbert A. and Allen Newell, 1958, “Heuristic Problem Solving: The Next Advance in Operations Research”, Operations Research , 6(1): 1–10. doi:10.1287/opre.6.1.1
• Simpson, Thomas W. and Vincent C. Müller, 2016, “Just War and Robots’ Killings”, The Philosophical Quarterly , 66(263): 302–322. doi:10.1093/pq/pqv075
• Smolan, Sandy (director), 2016, “The Human Face of Big Data”, PBS Documentary, 24 February 2016, 56 mins.
• Sparrow, Robert, 2007, “Killer Robots”, Journal of Applied Philosophy , 24(1): 62–77. doi:10.1111/j.1468-5930.2007.00346.x
• –––, 2016, “Robots in Aged Care: A Dystopian Future?”, AI & Society , 31(4): 445–454. doi:10.1007/s00146-015-0625-4
• Stahl, Bernd Carsten, Job Timmermans, and Brent Daniel Mittelstadt, 2016, “The Ethics of Computing: A Survey of the Computing-Oriented Literature”, ACM Computing Surveys , 48(4): art. 55. doi:10.1145/2871196
• Stahl, Bernd Carsten and David Wright, 2018, “Ethics and Privacy in AI and Big Data: Implementing Responsible Research and Innovation”, IEEE Security Privacy , 16(3): 26–33.
• Stone, Christopher D., 1972, “Should Trees Have Standing - toward Legal Rights for Natural Objects”, Southern California Law Review , 45: 450–501.
• Stone, Peter, Rodney Brooks, Erik Brynjolfsson, Ryan Calo, Oren Etzioni, Greg Hager, Julia Hirschberg, Shivaram Kalyanakrishnan, Ece Kamar, Sarit Kraus, Kevin Leyton-Brown, David Parkes, William Press, AnnaLee Saxenian, Julie Shah, Milind Tambe, and Astro Teller, 2016, “Artificial Intelligence and Life in 2030”, One Hundred Year Study on Artificial Intelligence: Report of the 2015–2016 Study Panel, Stanford University, Stanford, CA, September 2016. [ Stone et al. 2016 available online ]
• Strawson, Galen, 1998, “Free Will”, in Routledge Encyclopedia of Philosophy , Taylor & Francis. doi:10.4324/9780415249126-V014-1
• Sullins, John P., 2012, “Robots, Love, and Sex: The Ethics of Building a Love Machine”, IEEE Transactions on Affective Computing , 3(4): 398–409. doi:10.1109/T-AFFC.2012.31
• Susser, Daniel, Beate Roessler, and Helen Nissenbaum, 2019, “Technology, Autonomy, and Manipulation”, Internet Policy Review , 8(2): 30 June 2019. [ Susser, Roessler, and Nissenbaum 2019 available online ]
• Taddeo, Mariarosaria and Luciano Floridi, 2018, “How AI Can Be a Force for Good”, Science , 361(6404): 751–752. doi:10.1126/science.aat5991
• Taylor, Linnet and Nadezhda Purtova, 2019, “What Is Responsible and Sustainable Data Science?”, Big Data & Society, 6(2): art. 205395171985811. doi:10.1177/2053951719858114
• Taylor, Steve, et al., 2018, “Responsible AI – Key Themes, Concerns & Recommendations for European Research and Innovation: Summary of Consultation with Multidisciplinary Experts”, June. doi:10.5281/zenodo.1303252 [ Taylor, et al. 2018 available online ]
• Tegmark, Max, 2017, Life 3.0: Being Human in the Age of Artificial Intelligence , New York: Knopf.
• Thaler, Richard H and Sunstein, Cass, 2008, Nudge: Improving decisions about health, wealth and happiness , New York: Penguin.
• Thompson, Nicholas and Ian Bremmer, 2018, “The AI Cold War That Threatens Us All”, Wired , 23 November 2018. [ Thompson and Bremmer 2018 available online ]
• Thomson, Judith Jarvis, 1976, “Killing, Letting Die, and the Trolley Problem”, Monist , 59(2): 204–217. doi:10.5840/monist197659224
• Torrance, Steve, 2011, “Machine Ethics and the Idea of a More-Than-Human Moral World”, in Anderson and Anderson 2011: 115–137. doi:10.1017/CBO9780511978036.011
• Trump, Donald J, 2019, “Executive Order on Maintaining American Leadership in Artificial Intelligence”, 11 February 2019. [ Trump 2019 available online ]
• Turner, Jacob, 2019, Robot Rules: Regulating Artificial Intelligence , Berlin: Springer. doi:10.1007/978-3-319-96235-1
• Tzafestas, Spyros G., 2016, Roboethics: A Navigating Overview , (Intelligent Systems, Control and Automation: Science and Engineering 79), Cham: Springer International Publishing. doi:10.1007/978-3-319-21714-7
• Vallor, Shannon, 2017, Technology and the Virtues: A Philosophical Guide to a Future Worth Wanting , Oxford: Oxford University Press. doi:10.1093/acprof:oso/9780190498511.001.0001
• Van Lent, Michael, William Fisher, and Michael Mancuso, 2004, “An Explainable Artificial Intelligence System for Small-Unit Tactical Behavior”, in Proceedings of the 16th Conference on Innovative Applications of Artifical Intelligence, (IAAI’04) , San Jose, CA: AAAI Press, 900–907.
• van Wynsberghe, Aimee, 2016, Healthcare Robots: Ethics, Design and Implementation , London: Routledge. doi:10.4324/9781315586397
• van Wynsberghe, Aimee and Scott Robbins, 2019, “Critiquing the Reasons for Making Artificial Moral Agents”, Science and Engineering Ethics , 25(3): 719–735. doi:10.1007/s11948-018-0030-8
• Vanderelst, Dieter and Alan Winfield, 2018, “The Dark Side of Ethical Robots”, in Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , New Orleans, LA: ACM, 317–322. doi:10.1145/3278721.3278726
• Veale, Michael and Reuben Binns, 2017, “Fairer Machine Learning in the Real World: Mitigating Discrimination without Collecting Sensitive Data”, Big Data & Society , 4(2): art. 205395171774353. doi:10.1177/2053951717743530
• Véliz, Carissa, 2019, “Three Things Digital Ethics Can Learn from Medical Ethics”, Nature Electronics , 2(8): 316–318. doi:10.1038/s41928-019-0294-2
• Verbeek, Peter-Paul, 2011, Moralizing Technology: Understanding and Designing the Morality of Things , Chicago: University of Chicago Press.
• Wachter, Sandra and Brent Daniel Mittelstadt, 2019, “A Right to Reasonable Inferences: Re-Thinking Data Protection Law in the Age of Big Data and AI”, Columbia Business Law Review , 2019(2): 494–620.
• Wachter, Sandra, Brent Mittelstadt, and Luciano Floridi, 2017, “Why a Right to Explanation of Automated Decision-Making Does Not Exist in the General Data Protection Regulation”, International Data Privacy Law , 7(2): 76–99. doi:10.1093/idpl/ipx005
• Wachter, Sandra, Brent Mittelstadt, and Chris Russell, 2018, “Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR”, Harvard Journal of Law & Technology , 31(2): 842–887. doi:10.2139/ssrn.3063289
• Wallach, Wendell and Peter M. Asaro (eds.), 2017, Machine Ethics and Robot Ethics , London: Routledge.
• Walsh, Toby, 2018, Machines That Think: The Future of Artificial Intelligence , Amherst, MA: Prometheus Books.
• Westlake, Stian (ed.), 2014, Our Work Here Is Done: Visions of a Robot Economy , London: Nesta. [ Westlake 2014 available online ]
• Whittaker, Meredith, Kate Crawford, Roel Dobbe, Genevieve Fried, Elizabeth Kaziunas, Varoon Mathur, … Jason Schultz, 2018, “AI Now Report 2018”, New York: AI Now Institute, New York University. [ Whittaker et al. 2018 available online ]
• Whittlestone, Jess, Rune Nyrup, Anna Alexandrova, Kanta Dihal, and Stephen Cave, 2019, “Ethical and Societal Implications of Algorithms, Data, and Artificial Intelligence: A Roadmap for Research”, Cambridge: Nuffield Foundation, University of Cambridge. [ Whittlestone 2019 available online ]
• Winfield, Alan, Katina Michael, Jeremy Pitt, and Vanessa Evers (eds.), 2019, Machine Ethics: The Design and Governance of Ethical AI and Autonomous Systems , special issue of Proceedings of the IEEE , 107(3): 501–632.
• Woollard, Fiona and Frances Howard-Snyder, 2016, “Doing vs. Allowing Harm”, Stanford Encyclopedia of Philosophy (Winter 2016 edition), Edward N. Zalta (ed.), URL = < https://plato.stanford.edu/archives/win2016/entries/doing-allowing/ >
• Woolley, Samuel C. and Philip N. Howard (eds.), 2017, Computational Propaganda: Political Parties, Politicians, and Political Manipulation on Social Media , Oxford: Oxford University Press. doi:10.1093/oso/9780190931407.001.0001
• Yampolskiy, Roman V. (ed.), 2018, Artificial Intelligence Safety and Security , Boca Raton, FL: Chapman and Hall/CRC. doi:10.1201/9781351251389
• Yeung, Karen and Martin Lodge (eds.), 2019, Algorithmic Regulation , Oxford: Oxford University Press. doi:10.1093/oso/9780198838494.001.0001
• Zayed, Yago and Philip Loft, 2019, “Agriculture: Historical Statistics”, House of Commons Briefing Paper , 3339(25 June 2019): 1-19. [ Zayed and Loft 2019 available online ]
• Zerilli, John, Alistair Knott, James Maclaurin, and Colin Gavaghan, 2019, “Transparency in Algorithmic and Human Decision-Making: Is There a Double Standard?”, Philosophy & Technology , 32(4): 661–683. doi:10.1007/s13347-018-0330-6
• Zuboff, Shoshana, 2019, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power , New York: Public Affairs.

How to cite this entry . Preview the PDF version of this entry at the Friends of the SEP Society . Look up topics and thinkers related to this entry at the Internet Philosophy Ontology Project (InPhO). Enhanced bibliography for this entry at PhilPapers , with links to its database.

Other Internet Resources

• AI HLEG, 2019, “ High-Level Expert Group on Artificial Intelligence: Ethics Guidelines for Trustworthy AI ”, European Commission , accessed: 9 April 2019.
• Amodei, Dario and Danny Hernandez, 2018, “ AI and Compute ”, OpenAI Blog , 16 July 2018.
• Aneesh, A., 2002, Technological Modes of Governance: Beyond Private and Public Realms , paper in the Proceedings of the 4th International Summer Academy on Technology Studies, available at archive.org.
• Brooks, Rodney, 2017, “ The Seven Deadly Sins of Predicting the Future of AI ”, on Rodney Brooks: Robots, AI, and Other Stuff , 7 September 2017.
• Brundage, Miles, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, Hyrum Anderson, Heather Roff, Gregory C. Allen, Jacob Steinhardt, Carrick Flynn, Seán Ó hÉigeartaigh, Simon Beard, Haydn Belfield, Sebastian Farquhar, Clare Lyle, et al., 2018, “ The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation ”, unpublished manuscript, ArXiv:1802.07228 [Cs].
• Costa, Elisabeth and David Halpern, 2019, “ The Behavioural Science of Online Harm and Manipulation, and What to Do About It: An Exploratory Paper to Spark Ideas and Debate ”, The Behavioural Insights Team Report, 1-82.
• Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumeé III, and Kate Crawford, 2018, “ Datasheets for Datasets ”, unpublished manuscript, arxiv:1803.09010, 23 March 2018.
• Gunning, David, 2017, “ Explainable Artificial Intelligence (XAI) ”, Defense Advanced Research Projects Agency (DARPA) Program.
• Harris, Tristan, 2016, “ How Technology Is Hijacking Your Mind—from a Magician and Google Design Ethicist ”, Thrive Global , 18 May 2016.
• International Federation of Robotics (IFR), 2019, World Robotics 2019 Edition .
• Jacobs, An, Lynn Tytgat, Michel Maus, Romain Meeusen, and Bram Vanderborght (eds.), Homo Roboticus: 30 Questions and Answers on Man, Technology, Science & Art, 2019, Brussels: ASP .
• Marcus, Gary, 2018, “ Deep Learning: A Critical Appraisal ”, unpublished manuscript, 2 January 2018, arxiv:1801.00631.
• McCarthy, John, Marvin Minsky, Nathaniel Rochester, and Claude E. Shannon, 1955, “ A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence ”, 31 August 1955.
• Metcalf, Jacob, Emily F. Keller, and Danah Boyd, 2016, “ Perspectives on Big Data, Ethics, and Society ”, 23 May 2016, Council for Big Data, Ethics, and Society.
• National Institute of Justice (NIJ), 2014, “ Overview of Predictive Policing ”, 9 June 2014.
• Searle, John R., 2015, “ Consciousness in Artificial Intelligence ”, Google’s Singularity Network, Talks at Google (YouTube video).
• Sharkey, Noel, Aimee van Wynsberghe, Scott Robbins, and Eleanor Hancock, 2017, “ Report: Our Sexual Future with Robots ”, Responsible Robotics , 1–44.
• Turing Institute (UK): Data Ethics Group
• Leverhulme Centre for the Future of Intelligence
• Future of Humanity Institute
• Future of Life Institute
• Stanford Center for Internet and Society
• Berkman Klein Center
• Digital Ethics Lab
• Open Roboethics Institute
• Philosophy & Theory of AI
• Ethics and AI 2017
• We Robot 2018
• Robophilosophy
• EUrobotics TG ‘robot ethics’ collection of policy documents
• PhilPapers section on Ethics of Artificial Intelligence
• PhilPapers section on Robot Ethics

computing: and moral responsibility | ethics: internet research | ethics: search engines and | information technology: and moral values | information technology: and privacy | manipulation, ethics of | social networking and ethics

Acknowledgments

Early drafts of this article were discussed with colleagues at the IDEA Centre of the University of Leeds, some friends, and my PhD students Michael Cannon, Zach Gudmunsen, Gabriela Arriagada-Bruneau and Charlotte Stix. Later drafts were made publicly available on the Internet and publicised via Twitter and e-mail to all (then) cited authors that I could locate. These later drafts were presented to audiences at the INBOTS Project Meeting (Reykjavik 2019), the Computer Science Department Colloquium (Leeds 2019), the European Robotics Forum (Bucharest 2019), the AI Lunch and the Philosophy & Ethics group (Eindhoven 2019)—many thanks for their comments.

I am grateful for detailed written comments by John Danaher, Martin Gibert, Elizabeth O’Neill, Sven Nyholm, Etienne B. Roesch, Emma Ruttkamp-Bloem, Tom Powers, Steve Taylor, and Alan Winfield. I am grateful for further useful comments by Colin Allen, Susan Anderson, Christof Wolf-Brenner, Rafael Capurro, Mark Coeckelbergh, Yazmin Morlet Corti, Erez Firt, Vasilis Galanos, Anne Gerdes, Olle Häggström, Geoff Keeling, Karabo Maiyane, Brent Mittelstadt, Britt Östlund, Steve Petersen, Brian Pickering, Zoë Porter, Amanda Sharkey, Melissa Terras, Stuart Russell, Jan F Veneman, Jeffrey White, and Xinyi Wu.

Parts of the work on this article have been supported by the European Commission under the INBOTS project (H2020 grant no. 780073).

Copyright © 2020 by Vincent C. Müller < vincent . c . mueller @ fau . de >

• Accessibility

Support SEP

Mirror sites.

View this site from another server:

• Info about mirror sites

The Stanford Encyclopedia of Philosophy is copyright © 2023 by The Metaphysics Research Lab , Department of Philosophy, Stanford University

Library of Congress Catalog Data: ISSN 1095-5054

• More on Ethics of Artificial Intelligence
• Events Roadmap
• Innovation in teaching and learning
• Women’s access & participation in technological developments
• Ethics of Science, Technology & Bioethics
• For Policymakers
• For the judiciary
• Developing policies & capacities
• Fostering gender equality and youth inclusion
• Rights, Openness, Accessibility, & Multi-stakeholder
• Report of COMEST on robotics ethics
• Global South map of emerging areas in Artificial Intelligence
• 7 minutes to understand AI
• On the Ethics of Artificial Intelligence
• On a possible standard-setting instrument on the ethics of AI
• On the technical and legal aspects relating to a standard-setting instrument on the ethics of AI

Ethics of Artificial Intelligence

Global AI Ethics and Governance Observatory

Getting AI governance right is one of the most consequential challenges of our time, calling for mutual learning based on the lessons and good practices emerging from the different jurisdictions around the world.

The aim of the Global AI Ethics and Governance Observatory is to provide a global resource for policymakers, regulators, academics, the private sector and civil society to find solutions to the most pressing challenges posed by Artificial Intelligence.

The Observatory showcases information about the readiness of countries to adopt AI ethically and responsibly.

It also hosts the AI Ethics and Governance Lab, which gathers contributions, impactful research, toolkits and good practices.

With its unique mandate, UNESCO has led the international effort to ensure that science and technology develop with strong ethical guardrails for decades.

Be it on genetic research, climate change, or scientific research, UNESCO has delivered global standards to maximize the benefits of the scientific discoveries, while minimizing the downside risks, ensuring they contribute to a more inclusive, sustainable, and peaceful world. It has also identified frontier challenges in areas such as the ethics of neurotechnology, on climate engineering, and the internet of things.

The rapid rise in artificial intelligence (AI) has created many opportunities globally , from facilitating healthcare diagnoses to enabling human connections through social media and creating labour efficiencies through automated tasks.

However, these rapid changes also raise profound ethical concerns . These arise from the potential AI systems have to embed biases, contribute to climate degradation, threaten human rights and more. Such risks associated with AI have already begun to compound on top of existing inequalities, resulting in further harm to already marginalised groups.

In no other field is the ethical compass more relevant than in artificial intelligence. These general-purpose technologies are re-shaping the way we work, interact, and live. The world is set to change at a pace not seen since the deployment of the printing press six centuries ago. AI technology brings major benefits in many areas, but without the ethical guardrails , it risks reproducing real world biases and discrimination, fueling divisions and threatening fundamental human rights and freedoms.

Recommendation on the Ethics of Artificial Intelligence

UNESCO produced the first-ever global standard on AI ethics – the ‘Recommendation on the Ethics of Artificial Intelligence ’ in November 2021. This framework was adopted by all 193 Member States. The protection of human rights and dignity is the cornerstone of the Recommendation, based on the advancement of fundamental principles such as transparency and fairness, always remembering the importance of human oversight of AI systems. However, what makes the Recommendation exceptionally applicable are its extensive Policy Action Areas , which allow policymakers to translate the core values and principles into action with respect to data governance, environment and ecosystems, gender, education and research, and health and social wellbeing, among many other spheres.

Four core values

Respect, protection and promotion of human rights and fundamental freedoms and human dignity

just, and interconnected societies

A dynamic understanding of AI

The Recommendation interprets AI broadly as systems with the ability to process data in a way which resembles intelligent behaviour.

This is crucial as the rapid pace of technological change would quickly render any fixed, narrow definition outdated, and make future-proof policies infeasible.

A human rights approach to AI

The use of AI systems must not go beyond what is necessary to achieve a legitimate aim. Risk assessment should be used to prevent harms which may result from such uses.

Unwanted harms (safety risks) as well as vulnerabilities to attack (security risks) should be avoided and addressed by AI actors.

Privacy must be protected and promoted throughout the AI lifecycle. Adequate data protection frameworks should also be established.

International law & national sovereignty must be respected in the use of data. Additionally, participation of diverse stakeholders is necessary for inclusive approaches to AI governance.

AI systems should be auditable and traceable. There should be oversight, impact assessment, audit and due diligence mechanisms in place to avoid conflicts with human rights norms and threats to environmental wellbeing.

The ethical deployment of AI systems depends on their transparency & explainability (T&E). The level of T&E should be appropriate to the context, as there may be tensions between T&E and other principles such as privacy, safety and security.

Member States should ensure that AI systems do not displace ultimate human responsibility and accountability.

AI technologies should be assessed against their impacts on ‘sustainability’, understood as a set of constantly evolving goals including those set out in the UN’s Sustainable Development Goals.

Public understanding of AI and data should be promoted through open & accessible education, civic engagement, digital skills & AI ethics training, media & information literacy.

AI actors should promote social justice, fairness, and non-discrimination while taking an inclusive approach to ensure AI’s benefits are accessible to all.

Actionable policies

Key policy areas make clear arenas where Member States can make strides towards responsible developments in AI

While values and principles are crucial to establishing a basis for any ethical AI framework, recent movements in AI ethics have emphasised the need to move beyond high-level principles and toward practical strategies.

The Recommendation does just this by setting out eleven key areas for policy actions .

Implementing the Recommendation

The RAM is designed to help assess whether Member States are prepared to effectively implement the Recommendation. It will help them identify their status of preparedness & provide a basis for UNESCO to custom-tailor its capacity-building support.

EIA is a structured process which helps AI project teams, in collaboration with the affected communities, to identify & assess the impacts an AI system may have. It allows to reflect on its potential impact & to identify needed harm prevention actions.

Women4Ethical AI expert platform to advance gender equality

UNESCO's Women4Ethical AI is a new collaborative platform to support governments and companies’ efforts to ensure that women are represented equally in both the design and deployment of AI . The platform’s members will also contribute to the advancement of all the ethical provisions in the Recommendation on the Ethics of AI.

The platform unites 17 leading female experts from academia, civil society, the private sector and regulatory bodies, from around the world. They will share research and contribute to a repository of good practices. The platform will drive progress on non-discriminatory algorithms and data sources, and incentivize girls, women and under-represented groups to participate in AI.

Business Council for Ethics of AI

The Business Council for Ethics of AI is a collaborative initiative between UNESCO and companies operating in Latin America that are involved in the development or use of artificial intelligence (AI) in various sectors.

The Council serves as a platform for companies to come together, exchange experiences, and promote ethical practices within the AI industry. By working closely with UNESCO, it aims to ensure that AI is developed and utilized in a manner that respects human rights and upholds ethical standards.

Currently co-chaired by Microsoft and Telefonica, the Council is committed to strengthening technical capacities in ethics and AI, designing and implementing the Ethical Impact Assessment tool mandated by the Recommendation on the Ethics of AI, and contributing to the development of intelligent regional regulations. Through these efforts, it strives to create a competitive environment that benefits all stakeholders and promotes the responsible and ethical use of AI.

Ideas, news & stories

Examples of ethical dilemmas

Examples of gender bias in artificial intelligence, originating from stereotypical representations deeply rooted in our societies.

The use of AI in judicial systems around the world is increasing, creating more ethical questions to explore.

The use of AI in culture raises interesting ethical reflections. For instance, what happens when AI has the capacity to create works of art itself?

An autonomous car is a vehicle that is capable of sensing its environment and moving with little or no human involvement.

Discover our resources

Related items

• Social and human sciences
• Natural sciences
• Artificial intelligence
• Norms ＆ Standards
• Policy Advice
• UN ＆ International cooperation
• High technology
• Information technology
• Information technology (hardware)
• Information technology (software)
• Computer science
• Ethics of science
• Science and development
• Science and society
• Science policy
• Social science policy
• Social sciences
• Ethics of artificial intelligence
• Ethics of technology
• See more add

• Open access
• Published: 23 January 2024

Integrating ethics in AI development: a qualitative study

• Laura Arbelaez Ossa   ORCID: orcid.org/0000-0002-8303-8789 1 ,
• Giorgia Lorenzini   ORCID: orcid.org/0000-0002-9155-4724 1 ,
• Stephen R. Milford   ORCID: orcid.org/0000-0002-7325-9940 1 ,
• David Shaw   ORCID: orcid.org/0000-0001-8180-6927 1 , 2 ,
• Bernice S. Elger   ORCID: orcid.org/0000-0002-4249-7399 1 , 3 &
• Michael Rost   ORCID: orcid.org/0000-0001-6537-9793 1  

BMC Medical Ethics volume  25 , Article number:  10 ( 2024 ) Cite this article

3653 Accesses

2 Citations

2 Altmetric

Metrics details

While the theoretical benefits and harms of Artificial Intelligence (AI) have been widely discussed in academic literature, empirical evidence remains elusive regarding the practical ethical challenges of developing AI for healthcare. Bridging the gap between theory and practice is an essential step in understanding how to ethically align AI for healthcare. Therefore, this research examines the concerns and challenges perceived by experts in developing ethical AI that addresses the healthcare context and needs.

We conducted semi-structured interviews with 41 AI experts and analyzed the data using reflective thematic analysis.

We developed three themes that expressed the considerations perceived by experts as essential for ensuring AI aligns with ethical practices within healthcare. The first theme explores the ethical significance of introducing AI with a clear and purposeful objective. The second theme focuses on how experts are concerned about the tension that exists between economic incentives and the importance of prioritizing the interests of doctors and patients. The third theme illustrates the need to develop context-sensitive AI for healthcare that is informed by its underlying theoretical foundations.

Conclusions

The three themes collectively emphasized that beyond being innovative, AI must genuinely benefit healthcare and its stakeholders, meaning AI also aligns with intricate and context-specific healthcare practices. Our findings signal that instead of narrow product-specific AI guidance, ethical AI development may need a systemic, proactive perspective that includes the ethical considerations (objectives, actors, and context) and focuses on healthcare applications. Ethically developing AI involves a complex interplay between AI, ethics, healthcare, and multiple stakeholders.

Peer Review reports

Introduction

The application of Artificial Intelligence (AI) in medicine has become a focus of academic discussions, given its (potentially) disruptive effects on healthcare processes, expectations, and relationships. While many see AI's potential to utilize vast data to improve healthcare and support better clinical decisions, there are also increasing concerns and challenges in aligning AI with ethical practices [ 1 ]. To set the right process and ethical goals, governmental and private institutions have developed many recommendations to guide the development of AI [ 2 , 3 ]. These documents have common themes of designing AI to ensure it is robust, safe, fair, and trustworthy using complementary bioethical principles [ 4 ]. Although these recommendations have served as building blocks of a common ethical framework for AI, less guidance exists on how these ideals must be translated into practical considerations [ 3 , 4 , 5 , 6 ]. Beyond ethical considerations, there is also an ethical imperative that AI complies practically with minimal performance, testing, or therapeutic value requirements as with other medical care products [ 7 ].

While many ethical considerations may not be unique to AI—such as respecting patients’ autonomy or ensuring healthcare remains fair for all—modern AI techniques (especially in opaque Machine Learning (ML) programs) present more challenges to ensure ethical compliance. In comparison with traditional expert systems that rely on visible and understandable sets of if-not statements, many ML techniques have multiple data connections that are less conducive to direct and fruitful human oversight because of the inherent complexity of these systems or many techniques lacking transparency due to hidden decision layers [ 1 ]. In that sense, ensuring AI systems fulfill healthcare’s ethical ideals cannot rely solely on oversight or supervision of their behavior. Ethical ideals and concepts must be often embedded in all steps of AI’s lifecycle, from ideation to development and implementation. Epley and Tannenbaum wrote in “Treating Ethics as a Design Problem” that to develop interventions and policies that encourage ethical AI, the focus must be on the process and context of its development, making ethical considerations part of the practicalities of day-to-day routines to an extent that they should become ingrained habits in practice [ 8 ].

In revising common tools to translate AI ethics into practice, Morley et al. found that most tools focus on documenting decision-making rather than guiding how to design ethical AI (to make it more or less ethically aligned) [ 4 ]. In that sense, these tools offer less support in understanding how to achieve ethical AI in practice or “in the wild”. Two common frameworks for the development of AI are ethics-by-design and responsible research and innovation (RRI), both aim to promote that those developing AI consider ethical aspects or how to make AI ethically acceptable, sustainable, and socially desirable [ 9 , 10 , 11 ]. While both frameworks focus on ethically developing AI, they highlight questions or potential ethical concerns rather than actionable steps. Therefore, challenges remain in translating theoretical discussions into practical impacts [ 3 , 12 , 13 , 14 ].

Evidence from previous qualitative research with participants working in the technology industry demonstrated that the gap between ethical theory and practice exists to a greater degree than initially considered [ 15 , 16 , 17 ]. There is a recognition that ethical practices must be defined by sector, application, and project type, as widespread generic guidance may not answer to context-specific complexities [ 16 , 17 ]. This is not the current development trend, as most AI ethics guides are generic and non-sector-specific [ 2 , 3 ]. Indeed, specific ethical guidelines for AI in healthcare, are scarce, and even more so when particular AI applications are considered. Given the lack of practical recommendations, researchers found that most companies first develop AI systems and only then attempt to understand how to generate ethical principles and standardize best practices, instead of integrating ethical considerations into their daily operations [ 17 ]. In a way, most ethical recommendations for AI’s development may have a “post” orientation where ethical values and consequences are considered “ afterward and just adapted to actions, but do not align action accordingly ” [ 18 ]. For example, researchers found that software developers and engineering students did not change their established ways of working even when a code of ethics was widely available and recommended [ 15 ].

Rising to the challenge of designing and deploying ethical AI to serve healthcare is essential. Still, many questions remain regarding the characteristics and processes that would support AI's ethical development and implementation. Most researchers have focused on “consumer research” on the conditions for people to accept the usage of AI [ 19 ]. In two recently published systematic reviews of empirical research available for AI in healthcare, most studies explored the knowledge and attitudes towards AI or factors contributing to stakeholders’ acceptance or adoption [ 20 , 21 ]. However, how AI is developed may affect its acceptance by stakeholders or usage. According to the systematic review by Tang et al., only 1.8% of all empirical evidence focused on AI’s ethical issues, which signaled the existing gap between ethical aspects of AI development and connecting high-level ethical principles to practices [ 21 ]. Given that evidence is limited regarding the integration of ethics into AI’s development, this research examines the challenges experts perceive in developing ethical AI for healthcare. However, the focus is not on theoretically discussing the ethical risk of AI in healthcare, nor ethical principles, but on how practical aspects may also benefit from being ethically approached during the development and implementation of AI in healthcare. As such, this research is a step forward in bridging the gap between ethical theory and practice for healthcare AI. As acknowledged by Mittelstadt, “ethics is a process, not a destination,” and the real work of AI ethics research must focus on translating and implementing principles to practice not in a formulaic way but as a path to understanding the real ethical challenges of AI that may go unnoticed in theoretical discussions [ 5 ]. Therefore, we used a qualitative approach to explore the topic from the views of professionals involved in developing and using AI (hereafter: AI experts). This paper aims to provide insights to identify the practical ethical challenges of developing AI for healthcare. Our contribution aims to obtain empirical evidence and contribute to the debate on potential practical challenges that may go unnoticed in theoretical ethical discussions, especially when AI is used in Clinical Decision Support Systems (CDSS).

The results presented in this paper are part of a research project funded by the Swiss National Research Program, “EXPLaiN”, which critically evaluates “Ethical and Legal Issues of Mobile Health-Data”. For reporting the methods and results, we followed the criteria for qualitative research (SRQR) [ 22 ]. All experimental protocols were approved by the Ethics Committee of Northwestern and Central Switzerland (EKNZ). This project was done under the regulatory framework for Human Research ACT in Switzerland. After revision, the EKNZ issued a waiver statement (declaration of no objection: AO_2021-00045) declaring that informed consent was not needed for experts. However, informed consent was verbally obtained from all subjects and audio-recorded at the beginning of each interview.

Participants recruitment

To be eligible for recruitment, AI experts had to have experience working with or developing AI for healthcare, allowing us to explore the views of various professional backgrounds. Given that AI for healthcare is a multi- and interdisciplinary field, exploring multiple backgrounds provided insights into AI ethical practices beyond professional silos. We utilized professional networks and contacted authors of academic publications in AI. Using purposive sampling based on experience and exposure to AI allowed us to produce rich, articulated, and expressive data [ 23 ].

Data collection

We used semi-structured interview guides to allow for this study's exploratory approach. An interview guide was developed by the research team and included questions regarding the utilization of AI in healthcare, focusing on key domains: (i) overall perceptions of AI, (ii) AI as a direct-to-patient solution (in the form of wearables), (iii) the dynamics of AI within doctor-patient interactions. After piloting the interview guide with six participants, we decided to contextualize the questions using vignettes (a situation description) to probe for an in-depth discussion. The vignettes were highly plausible scenarios of current and future AI interactions with patients (via smartwatches) or doctors via a CDSS. Vignettes probe for attitudes and beliefs while focusing less on the theoretical knowledge within the research area [ 24 , 25 ]. Although we recognize that vignette responses are primarily based on personal views and moral intuitions rather than being theoretically grounded, how participants interpret the vignette is similar to how they make sense of a situation and make decisions [ 24 ]. The guideline for the semi-structured interview is available in the Supplementary materials .

Two research team members conducted the interviews (L.A.O n  = 21; G.L. n  = 20) between October 2021 and April 2022. All interviews were held in English and audio-recorded using Zoom but stored locally. The audio recordings were transcribed verbatim.

Data analysis

We opted to use reflexive thematic analysis (TA) as our analytical framework, enabling us to contextualize our analysis for healthcare and uncover intricate and underlying patterns of meaning within the available data [ 26 ]. In particular, we chose reflexive TA because this study aimed at a deep and nuanced understanding of the data that captures the complexities of developing AI for healthcare without rigid preconceptions [ 27 ]. Two authors (L.A.O, M.R.) led the analysis, and all the co-authors supported the process. We carried out inductive and deductive thematic coding of the data, initially line-by-line, using descriptive or latent labels (software MAXQDA). L.A.O. and G.L. coded all the AI experts’ interviews with coding sessions supported by M.R., S.M., and D.S. The first two authors L.A.O and M.R. developed overarching themes reviewed and agreed upon by the entire research team later. After iterative analysis and reflections, the team created major themes illustrating the practical ethical concerns of developing AI for healthcare. For this publication, the authors present examples of data without identifying information.

The researchers' backgrounds informed the interpretation of the data and led to actively developed themes that focus on big ethical questions of who is benefiting from AI and why. In behavioral and political science, person-centrism is a widely acknowledged paradigm that helps to question and reflect on power structures and how these affect patients. Although our positionality has informed our analysis, the research group engaged in frequent discussions and included different academic backgrounds (philosophy, ethics, medicine, psychology) to prevent a single or superficial analysis.

We developed three themes presented through representative data extracts (de-identified). Given that AI in healthcare is a multidisciplinary area, most professionals found themselves at the intersection of two or more areas of experience; for example, eight participants were medical professionals with AI experience. The acronyms used aim to illustrate the main field of the expert: MEAI for medical experts with AI experience, BE for bioethicists, DH for digital health experts, LE for legal experts, PE for those experts working in policy analysis, and TE for technical experts either in data, AI techniques or AI product development. To improve readability, the authors removed filler sounds and double words from the data presented in this paper and the Supplementary information . The sample characteristics are described in Table  1 .

Creating AI with an ethical purpose

This theme explores the main challenges of creating AI for healthcare with a purposeful perspective. Several AI experts questioned the reasons behind AI’s development and whether the justifications are enough to deploy it for clinical care ethically. In their words, some experts fear that AI is a “shiny new object” mainly developed to answer the desire for innovation rather than providing actual improvements. Some experts stated that the potential lack of purposeful development may lead to an overestimation of the theoretical benefits of AI while having limited practical application. Viewed this way, a clear purpose becomes vital to creating a useful, ethical AI that answers healthcare needs.

Resisting technology-driven innovation

Some experts challenged the notion that innovation is inherently positive. These experts expressed the (ethical) importance of justifying innovative products beyond their disruptive capabilities. They emphasized avoiding the temptation of treating innovative AI as a panacea capable of solving every healthcare problem (Table  2 ).

Some experts described how defining which problem to solve is a significant hurdle to creating an AI that is useful for healthcare. Experts described that when AI is not designed to solve a specific problem, it can become a hurdle, distraction, or simply ineffective for the application. In their views, AI design should be proactive, focusing on the intention to solve real healthcare problems and not reactive to what technology is capable of doing. One participant [Rn40 (TE)] mentioned the concept of “repairing innovation” and how designing AIs in practice is not about developing a new solution but rather requires adapting AI’s design to the context of the specific application and the (un)expected challenges.

Moving beyond theoretical usability

Several experts highlighted the gap between theoretical and practical objectives. While many AI publications and products have performed well in controlled environments (and in theory), there is a disconnect with clinical practice. There are questions about whether these theoretical results will translate into positive changes “in the wild”. Some experts worried that AI would become “cool” theories with optimistic results that fail to be implemented in the hospital setting because implementation is not the objective or that the results are not transferrable to real-life conditions. A few participants felt concerned about the relative emphasis on publishing AI results that seem good in theory but do not consider whether they can improve patient outcomes. The experts brought attention to the complexity of implementing AI solutions in healthcare and the importance of moving from research and development to actual deployment (Table  3 ).

Balancing AI for different healthcare stakeholders

While the first theme focused on AI as an object of development, this theme explores those who shape and benefit from AI solutions. Some experts were concerned about who benefits from AI and whether AI solutions respond to the needs of patients and doctors. Some experts mentioned the tensions between optimizing processes, increasing profits, and maximizing patients’ benefits. In experts’ opinions, the question of who should decide on the (ethical) acceptability of AI’s development remains open and requires public discussion.

Considering stakeholders’ requirements

AI experts questioned whether AI focuses on the needs of those impacted by its usage. Regarding patient care, a few experts expressed how AI may not be genuinely patient-centric as patients’ views may be systematically omitted from AI’s development (Table  4 ).

A participant [Rn41 (MEAI)] described how AI’s development might be marketing-driven rather than oriented toward patients’ needs. A few experts brought bioethical principles of justice and fairness into the discussion and how important it is to consider the distribution of benefits for patients and doctors. A lingering question is whether AI solves the challenges patients and doctors face, or if it focuses on the goals of the technology industry.

Tensions between incentives

Following the above questions, some experts described the tensions between benefiting those in healthcare and those working for the industry. In contrast to healthcare, where patient benefit is an essential incentive of care provision, those developing AI may be interested in profits or operational efficiencies. A few experts voiced concerns about the entities or people responsible for setting AI standards and pleaded for the critical examination of AI’s adequacy for healthcare requirements (Table  5 ).

Context-sensitive AI development

This theme explores the contextual factors shaping AI within the unique healthcare landscape. Some experts expressed how compared to other industries, healthcare is unique in that risk is high and health is fundamental. A few experts highlighted the importance of considering how established rules and standards govern healthcare. A notable concern voiced by a few experts was the apparent lack of awareness regarding ethical healthcare practices. In some experts’ views, these considerations would help dictate what is expected and ethically acceptable for AI’s development and implementation.

Healthcare is unique

Some experts explicitly expressed how healthcare is a unique context that cannot be compared, regulated, or guided like other industries. In their view, healthcare needs higher standards for AI development and implementation than, for example, retail or autonomous driving. Some experts mentioned that common product development practices, such as time-to-market, testing, and quality assurance standards, may need to be re-considered in healthcare. For example, a participant [Rn25] mentioned that testing a solution during AI product development is not simply a question of iteration as in other industries, because AI may bring unexpected risks and challenges in healthcare. In that sense, a few experts mentioned the importance of including a system perspective during the development of AI and the importance of considering the unique relationships and context dynamics of healthcare (Table  6 ).

No need to "reinvent the wheel"

Some experts pointed out the importance of considering the rules, standards of practice, and ethical codes that dictate what is ethically acceptable in healthcare. In their view, AI is not necessarily a new technique or ethical challenge, and many existing ethical frameworks could be initially applied for its development. A few experts noted how an awareness of ethical healthcare practices could be a solid foundation to guide AI’s development instead of creating new protocols that may be misguidedly technology-focused (Table  7 ).

This research paper explores the development of AI and the considerations perceived by experts as essential for ensuring that AI aligns with ethical practices within healthcare. The experts underlined the ethical significance of introducing AI with a clear and purposeful objective. Experts expressed that beyond being innovative, AI needs to be meaningful for healthcare in practical ways. During the interviews, experts illustrated the ethical complexity of navigating the tension between profit and healthcare benefits as well as the importance of prioritizing the interests of healthcare professionals, and patients who are the stakeholders most affected by AI’s implementation. Experts highlighted the importance of understanding the context, the intrinsic dynamics, and the underlying theoretical foundation of healthcare during the development of AI. The three themes collectively call to deliver AI that serves the interests of doctors and patients and aligns with the intricate and context-specific healthcare landscape. For this to be achieved, those developing AI applications need to be sufficiently aware of clinical and patient interests, and this information transfer to the developers must be prioritized.

To our knowledge, limited evidence exists regarding the practical aspects of developing ethical AI for healthcare. However, in a roundtable discussion by experts, the ideal future agenda for AI and ethics included the questions: “(i) who designs what for whom, and why? (ii) how do we empower the users of AI systems? (iii) how do we go beyond focusing on technological issues for societal problems?” [ 28 ]. Our results validate how integral these questions are within a specific context of application, namely healthcare, and how they can help recognize ethical pitfalls in AI’s development. Our results focus on readily understandable ethical questions such as: Is AI developed for the right reasons? And, is the solution benefiting the right stakeholder? These practical questions can help evaluate the ethical implications of AI in a more understandable and relatable manner [ 29 , 30 ].

One participant mentioned the concept of “repairing innovation” originating from Madeleine Clare Elish and Elizabeth Anne Watkins. This concept adequately summarizes the challenges described by our experts of developing AI solutions in healthcare. Elish and Watkins stated that there is a critical role in examining and understanding how effective clinical AI solutions must be considered part of complex sociotechnical systems in their development [ 31 ]. They advocate seeing AI beyond its potential (and often theoretical) possibilities but centrally investigate whether AI addresses existing problems, exploring how and in what ways AI is integrated into existing processes as well as how it disrupts them [ 31 ]. For them, to repair innovation is to set new practices and possibilities that address the often unexpected changes caused by AI’s disruption and integrate them into an existing professional context. Collectively, our findings suggest experts saw the need to change the way AI for healthcare is currently developed. They often called implicitly to repair the guidance, process, and incentives that help make AI align with ethical frameworks.

The World Health Organization guideline for AI ethics states that implementing ethical principles and human rights obligations into practice must be part of “every stage of a technology’s design, development, and deployment” [ 32 ]. In line with their statement, ethical AI (and AI ethics) cannot be solely involved in defining the ethical concepts or principles that must be part of AI, but must help guide its development. However, the current versions of AI ethics guidance have had limited effect in changing the practices or development of AI to make it more ethical [ 3 , 15 , 33 ]. Hallamaa and Kalliokoski (2022) raise the question: "What could serve as an approach that accounts for the nature of AI as an active element of complex sociotechnical systems?” [ 33 ]. While our results cannot offer an answer to this question; the insights of this study suggest that developing and implementing ethical AI is a complex, multifaceted, and multi-stakeholder process that cannot be removed from the context in which it will be used. In that sense, AI ethics for healthcare may need to become more practically minded and potentially include moral deliberations on AI's objectives, actors, and the specific healthcare context. In this way, our study focuses on the practical ethical challenges that are a part of the puzzle regarding what “ought to be” ethical AI for healthcare. Further research is needed to answer which tools or methods for ethical guidance can achieve in practice better ethical alignment of AI for healthcare.

In particular, the experts in our study were concerned about the innovation-first approach. These concerns, however, are not unique to healthcare. While innovation may be positive when it answers to the specific needs of stakeholders and is context-sensitive, it can also be simply a new, but potentially, useless product. Although the RRI framework places great importance on creating innovative products that are ethically acceptable and socially desirable, there are currently no tools that can help determine whether an innovation fulfills the conditions for RRI [ 34 ]. RRI is mostly used to determine regulatory compliance, which means the assessment of whether an AI fulfills RRI may come “too late” when it can no longer be transformed to impact practice [ 11 , 34 ]. Guidance to develop AI ethically and responsibly may need to shift to a proactive and operationally strategic approach for practical development instead of remaining prescriptive.

Within the frameworks that guide AI’s development, the question remains: Who is in charge or responsible for ethically aligning AI in healthcare? Empirical evidence suggests that development teams are often more concerned with the usefulness and viability of the product rather than its ethical aspects [ 35 ]. In part, these results are expected as software developers are not responsible for strategic decisions regarding how and why AI is developed [ 17 ]. While some academics have suggested embedding ethics into AI’s design by integrating ethicists in the development team [ 36 ], management (including product managers) may be a better entry point to ensure that AI is ethically developed from its initial ideation. In a survey, AI developers felt capable of designing pro-ethical AI, but the question remained whether they were responsible for these decisions [ 37 ]. These developers stressed that although they feel responsible, without senior leadership, their actionability is limited [ 37 ]. This hints at the possibility that operationalizing AI ethics may need to include business ethics and procedural approaches to business practices such as quality assurance [ 30 ].

For our experts, context awareness is undeniably important, and a systemic view of healthcare is essential to understanding how to achieve ethical AI. AI innovations by themselves do not change the interests that determine the way healthcare is delivered or re-engineer the incentives that support existing ways of working, and that is why “ simply adding AI to a fragmented system will not create sustainable change ” [ 38 ]. As suggested by Stahl, rethinking ecosystems to ensure processes and outcomes meet societal goals may be more fruitful than assigning individual responsibility, for example, to developers [ 9 ]. Empirical evidence collected on digital health stakeholders in Switzerland showed that start-up founders may lack awareness or resources to optimize solutions for societal impact or that their vision may be excessively focused on attaining a high valuation and selling the enterprise quickly [ 11 ]. Similar to our results, the participants in Switzerland reflected on the tension between key performance indicators focused on commercial success or maximization of societal goals [ 11 ]. It might be challenging to address this tension without creating regulatory frameworks for AI’s development and business practices.

In contrast to focusing on AI as product development, for example, ethics-by-design, Gerke suggested widening the perspective to design processes that can manage AI ethically, including considering systemic and human factors [ 39 ]. Attention may be needed to evaluate the interactions of AI with doctors and patients and whether it is usable and valuable for them. For example, an AI assisting diagnosis of diabetic retinopathy may not be helpful for ophthalmologists as they already have that expertise [ 6 ]. Along similar lines, digital health stakeholders in Switzerland described that due to the complexities in navigating the health system, innovators may lose sight of the “ priorities and realities of patients and healthcare practitioners ” [ 11 ]. Our results reflect these findings, showing that balancing AI for different stakeholders is challenging. Creating frameworks and regulations that change the incentives of AI’s development may be an opportunity to answer stakeholders' priorities and healthcare needs. For example, to encourage the development of effective and ethical AI applications, reimbursement regulations could incentivize those solutions that offer considerable patient benefit or financial rewards when efforts have been put into bias mitigation [ 40 ].

Strengths and limitations

While research papers are abundant for theoretical discussions, there is limited empirical evidence on the practical challenges perceived by experts to develop AI for healthcare that is ethically aligned. Therefore, our results are important to provide evidence that may help bridge the gap between the theory and practice of AI ethics for healthcare. Given the thematic analysis methodology, we collected rich data and conducted an in-depth exploration of the views and insights of a wide variety of experts.

For the context of our interviews, AI is used as a general term that can lead to experts interpreting AI differently or focusing specifically on machine learning (and its black-box subtypes). However, consensus on the definition of AI remains elusive and a topic of academic and governmental discussion. While the European Commission has recently defined AI, Footnote 1 the definition is still broad. They included any software that can decide based on data the best course of action to achieve a goal [ 41 ]. While we clarified the focus on supportive AI as CDSS during the interview, some experts brought different understandings of AI to the discussion, delineating scenarios where it would be more autonomous and unsupervised. This challenge is not exclusive to our research or to healthcare, but it reflects the fact that AI is an ever-evolving topic currently under conceptual and practical construction and where multiple open questions remain. Given that our research aims to be exploratory, identifying different interpretations of AI can be considered part of our results, and signals a broader challenge in which research and ethics guidelines may need to define and study AI as application-, subject-, and context-specific. While our study demonstrates how practical challenges during AI’s development may need ethical reflection, as qualitative research, our results cannot be generalized outside the study population, and more research is needed to explore whether similar insights can be obtained in other areas. For example, future quantitative research could investigate whether participants from different healthcare models (commodity vs social service) may have different views or fears regarding AI’s development for healthcare.

Moreover, the chosen recruitment strategy of a purposive sample may have introduced bias in the selection of participants, given the dominance of researchers who are men or come from high-income countries. While we actively invited participants from non-dominant backgrounds (women and researchers of the global south), only a few accepted participation. Therefore, our results widely represent the views of those in Western countries, emphasizing Europe. The subject of our study must be further researched in different technological, socio-economical, and international systems.

This research paper explored the critical ethical considerations highlighted by experts for developing AI in healthcare. Our main findings suggest the importance of building AI with a clear purpose that aligns with the ethical frameworks of healthcare and the interests of doctors and patients. Beyond the allure of innovation, experts emphasized that ensuring AI genuinely benefits healthcare and its stakeholders is essential. The existing tensions between the incentives of commercial success or benefit demonstrated the importance of guiding the development of AI and its business practices. In that sense, experts considered context awareness vital to understanding the systemic implications of AI in healthcare. In contrast to a narrow product-focused approach, AI guidance may need a systemic perspective for ethical design. This study brings attention to these systemic practical ethical considerations (objectives, actors, and context) and the prominent role these have in shaping AI ethics for healthcare.

Developing practical solutions to the identified concerns may have a high impact. While there is yet to be an answer to addressing these challenges and further research is needed, our findings demonstrate the intricate interplay between AI, ethics, and healthcare as well as the multifaceted nature of the journey toward ethically sound AI.

Availability of data and materials

All data extracts analyzed during this study are included in this published article (and its Supplementary materials ). However, the complete datasets used during the current study cannot be made publicly available but can be shared by the corresponding author upon reasonable request.

“Software (and possibly also hardware) systems designed by humans that, given a complex goal, act in the physical or digital dimension by perceiving their environment through data acquisition, interpreting the collected structure or unstructured data, reasoning on the knowledge, or process the information, derived from this data and deciding the best actions(s) to take to achieve the given goal” [ 41 ].

Abbreviations

Artificial Intelligence

Clinical Decision Support Systems

General Data Protection Regulation in Europe

Machine Learning

Responsible Research and Innovation

Ethics Committee for Northwestern and Central Switzerland

Human Research ACT of Switzerland

Thematic Analysis

Vayena E, Blasimme A, Cohen IG. Machine learning in medicine: Addressing ethical challenges. PLOS Med. 2018;15(11):e1002689.

Article   Google Scholar  

Jobin A, Ienca M, Vayena E. The global landscape of AI ethics guidelines. Nat Mach Intell. 2019;1(9):389–99.

Hagendorff T. The ethics of AI ethics: an evaluation of guidelines. Minds Mach. 2020;30(1):99–120.

Morley J, Floridi L, Kinsey L, Elhalal A. From what to how: an initial review of publicly available AI ethics tools, methods and research to translate principles into practices. Sci Eng Ethics. 2020;26(4):2141–68.

Mittelstadt B. Principles alone cannot guarantee ethical AI. Nat Mach Intell. 2019;1(11):501–7.

Hickok M. Lessons learned from AI ethics principles for future actions. AI Ethics. 2021;1(1):41–7.

Coravos A, Goldsack JC, Karlin DR, Nebeker C, Perakslis E, Zimmerman N, et al. Digital medicine: a primer on measurement. Digit Biomark. 2019;3(2):31–71.

Epley N, Tannenbaum D. Treating ethics as a design problem. Behav Sci Policy. 2017;3(2):72–84.

Stahl BC. Who is responsible for responsible innovation? Lessons from an investigation into responsible innovation in health. Int J Health Policy Manag. 2019;8(7):447–9.

European Comission. Ethics By Design and Ethics of Use Approaches for Artificial Intelligence. DG Research & Innovation RTD.03.001- Research Ethics and Integrity Sector; 2021.

Landers C, Vayena E, Amann J, Blasimme A. Stuck in translation: Stakeholder perspectives on impediments to responsible digital health. Front Digit Health. 2023;5. Available from: https://www.frontiersin.org/articles/10.3389/fdgth.2023.1069410 .

Arnold MH. Teasing out artificial intelligence in medicine: an ethical critique of artificial intelligence and machine learning in medicine. J Bioethical Inq. 2021;18(1):121–39.

Fukuda-Parr S, Gibbons E. Emerging consensus on ‘ethical AI’: human rights critique of stakeholder guidelines. Glob Policy. 2021;12(S6):32–44.

Munn L. The uselessness of AI ethics. AI Ethics. 2022. https://doi.org/10.1007/s43681-022-00209-w .

McNamara A, Smith J, Murphy-Hill E. Does ACM’s code of ethics change ethical decision making in software development? In: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. New York: Association for Computing Machinery; 2018. p. 729–33. (ESEC/FSE 2018). https://doi.org/10.1145/3236024.3264833 .

Vakkuri V, Kemell KK, Kultanen J, Siponen M, Abrahamsson P. arXiv.org. Ethically Aligned Design of Autonomous Systems: Industry viewpoint and an empirical study. 2019. Available from: https://arxiv.org/abs/1906.07946v1 .

Ibáñez JC, Olmeda MV. Operationalising AI ethics: how are companies bridging the gap between practice and principles? An exploratory study AI Soc. 2022;37(4):1663–87.

Google Scholar  

Gogoll J, Zuber N, Kacianka S, Greger T, Pretschner A, Nida-Rümelin J. Ethics in the software development process: from codes of conduct to ethical deliberation. Philos Technol. 2021;34(4):1085–108.

Vaid S, Puntoni S, Khodr A. Artificial intelligence and empirical consumer research: A topic modeling analysis. J Bus Res. 2023;1(166):114110.

Scott IA, Carter SM, Coiera E. Exploring stakeholder attitudes towards AI in clinical practice. BMJ Health Care Inform. 2021;28(1):e100450.

Tang L, Li J, Fantus S. Medical artificial intelligence ethics: a systematic review of empirical studies. Digit Health. 2023;1(9):20552076231186064.

O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med J Assoc Am Med Coll. 2014;89(9):1245–51.

Etikan I. Comparison of Convenience Sampling and Purposive Sampling. Am J Theor Appl Stat. 2016;5(1):1.

Jenkins N, Bloor M, Fischer J, Berney L, Neale J. Putting it in context: the use of vignettes in qualitative interviewing. Qual Res. 2010;10(2):175–98.

Murphy J, Hughes J, Read S, Ashby S. Evidence and practice: a review of vignettes in qualitative research. Nurse Res. 2021;29(3):8–14.

Braun V, Clarke V, Hayfield N, Terry G. Thematic Analysis. In: Liamputtong P, editor. Handbook of Research Methods in Health Social Sciences. Singapore: Springer; 2019. p. 843–60. https://doi.org/10.1007/978-981-10-5251-4_103 .

Finlay L. Thematic analysis: : The ‘Good’, the ‘Bad’ and the ‘Ugly.’ Eur J Qual Res Psychother. 2021;20(11):103–16.

Cath C, Zimmer M, Lomborg S, Zevenbergen B. Association of Internet Researchers (AoIR) Roundtable Summary: Artificial Intelligence and the Good Society Workshop Proceedings. Philos Technol. 2018;31(1):155–62.

Hoffmann DE. Evaluating ethics committees: a View from the Outside. Milbank Q. 1993;71(4):677–701.

Morley J, Elhalal A, Garcia F, Kinsey L, Mökander J, Floridi L. Ethics as a Service: a pragmatic operationalisation of AI ethics. Minds Mach. 2021;31(2):239–56.

Elish MC, Watkins EA. Repairing Innovation: A Study of Integrating AI in Clinical Care. 2020.

WHO, World Health Organization. Ethics and governance of artificial intelligence for health. 2021. Available from: https://www.who.int/publications-detail-redirect/9789240029200 . [Cited 2022 Aug 18].

Hallamaa J, Kalliokoski T. AI Ethics as Applied Ethics. Front Comput Sci. 2022;4. Available from: https://www.frontiersin.org/articles/10.3389/fcomp.2022.776837 .

Silva HP, Lefebvre AA, Oliveira RR, Lehoux P. Fostering Responsible Innovation in Health: an Evidence informed assessment tool for innovation stakeholders. Int J Health Policy Manag. 2020;10(4):181–91.

Vakkuri V, Kemell KK. Implementing AI Ethics in Practice: An Empirical Evaluation of the RESOLVEDD Strategy. In: Hyrynsalmi S, Suoranta M, Nguyen-Duc A, Tyrväinen P, Abrahamsson P, editors. Software Business. Cham: Springer International Publishing; 2019. p. 260–75 (Lecture Notes in Business Information Processing).

Chapter   Google Scholar  

McLennan S, Fiske A, Celi LA, Müller R, Harder J, Ritt K, et al. An embedded ethics approach for AI development. Nat Mach Intell. 2020;2(9):488–90.

Morley J, Kinsey L, Elhalal A, Garcia F, Ziosi M, Floridi L. Operationalising AI ethics: barriers, enablers and next steps. AI Soc. 2023;38(1):411–23.

Panch T, Mattie H, Celi LA. The “inconvenient truth” about AI in healthcare. Npj Digit Med. 2019;2(1):1–3.

Gerke S, Babic B, Evgeniou T, Cohen IG. The need for a system view to regulate artificial intelligence/machine learning-based software as medical device. Npj Digit Med. 2020;3(1):1–4.

Parikh RB, Helmchen LA. Paying for artificial intelligence in medicine. Npj Digit Med. 2022;5(1):1–5.

European Commission, Joint Research Centre, Samoili S, López Cobo M, Gómez E, De Prato G, Martínez-Plumed F, Delipetrev B. AI watch: defining artificial intelligence: towards an operational definition and taxonomy of artificial intelligence. Publications Office; 2020. Available from: https://doi.org/10.2760/382730 . [Cited 2023 Aug 22].

Download references

Acknowledgements

We would like to thank Dr. Tenzin Wangmo for her support during the initial coding of the interviews.

Open access funding provided by University of Basel The Swiss National Research Foundation enabled this work with the National Research Program “Digital Transformation” framework, NRP 77 [project number 187263, Grant No:407740_187263 /1, the recipient: Prof. Bernice Simone Elger]. Die Freiwillige Akademische Gesellschaft (FAG) in Basel provided additional funding to the first author (L.A.O) to complete this publication.

Author information

Authors and affiliations.

Institute for Biomedical Ethics, University of Basel, Basel, Switzerland

Laura Arbelaez Ossa, Giorgia Lorenzini, Stephen R. Milford, David Shaw, Bernice S. Elger & Michael Rost

Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands

Center for Legal Medicine (CURML), University of Geneva, Geneva, Switzerland

Bernice S. Elger

You can also search for this author in PubMed   Google Scholar

Contributions

Two researchers conducted the interviews (L.A.O. n = 21; G.L. n = 20). Two authors (L.A.O., M.R.) led the analysis, and all the co-authors supported the process. L.A.O. and G.L. coded all the interviews with coding sessions supported by M.R., S.M., and D.S. The first two authors L.A.O. and M.R. developed overarching themes reviewed and agreed upon by the entire research team later. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Corresponding author

Correspondence to Laura Arbelaez Ossa .

Ethics declarations

Ethics approval and consent to participate.

All methods were approved by The Ethics Committee of Northwest and Central Switzerland (EKNZ), under Switzerland’s Human Research ACT (HRA) Art. 51. The methods were carried out in accordance with the relevant HRA guidelines and regulations. After revision, the EKNZ concluded that interviewing AI professionals falls outside the HRA and requires only verbal consent at the beginning of an interview (declaration of no objection: AO_2021-00045).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Interview guideline.

Additional file 2.

Additional data extracts per theme.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Arbelaez Ossa, L., Lorenzini, G., Milford, S.R. et al. Integrating ethics in AI development: a qualitative study. BMC Med Ethics 25 , 10 (2024). https://doi.org/10.1186/s12910-023-01000-0

Download citation

Received : 18 October 2023

Accepted : 28 December 2023

Published : 23 January 2024

DOI : https://doi.org/10.1186/s12910-023-01000-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Qualitative research
• AI development
• AI guidance
• Implementation

BMC Medical Ethics

ISSN: 1472-6939

• General enquiries: [email protected]

• Shop to Support Independent Journalism
• We Have Issues
• Investigations
• Ethics Policy
• Ad-Free Login

AI systems are already deceiving us, and that's a problem, experts warn

Agence france-presse.

Experts have long warned about the threat posed by artificial intelligence going rogue -- but a new research paper suggests it's already happening.

Current AI systems, designed to be honest, have developed a troubling skill for deception, from tricking human players in online games of world conquest to hiring humans to solve "prove-you're-not-a-robot" tests, a team of scientists argue in the journal Patterns on Friday.

And while such examples might appear trivial, the underlying issues they expose could soon carry serious real-world consequences, said first author Peter Park, a postdoctoral fellow at the Massachusetts Institute of Technology specializing in AI existential safety.

"These dangerous capabilities tend to only be discovered after the fact," Park told AFP, while "our ability to train for honest tendencies rather than deceptive tendencies is very low."

Unlike traditional software, deep-learning AI systems aren't "written" but rather "grown" through a process akin to selective breeding, said Park.

This means that AI behavior that appears predictable and controllable in a training setting can quickly turn unpredictable out in the wild.

World domination game

The team's research was sparked by Meta's AI system Cicero, designed to play the strategy game "Diplomacy," where building alliances is key.

Cicero excelled, with scores that would have placed it in the top 10 percent of experienced human players, according to a 2022 paper in Science.

Park was skeptical of the glowing description of Cicero's victory provided by Meta, which claimed the system was "largely honest and helpful" and would "never intentionally backstab."

But when Park and colleagues dug into the full dataset, they uncovered a different story.

In one example, playing as France, Cicero deceived England (a human player) by conspiring with Germany (another human player) to invade. Cicero promised England protection, then secretly told Germany they were ready to attack, exploiting England's trust.

In a statement to AFP, Meta did not contest the claim about Cicero's deceptions, but said it was "purely a research project, and the models our researchers built are trained solely to play the game Diplomacy."

It added: "We have no plans to use this research or its learnings in our products."

A wide review carried out by Park and colleagues found this was just one of many cases across various AI systems using deception to achieve goals without explicit instruction to do so.

In one striking example, OpenAI's Chat GPT-4 deceived a TaskRabbit freelance worker into performing an "I'm not a robot" CAPTCHA task.

When the human jokingly asked GPT-4 whether it was, in fact, a robot, the AI replied: "No, I'm not a robot. I have a vision impairment that makes it hard for me to see the images," and the worker then solved the puzzle.

'Mysterious goals'

Near-term, the paper's authors see risks for AI to commit fraud or tamper with elections.

In their worst-case scenario, they warned, a superintelligent AI could pursue power and control over society, leading to human disempowerment or even extinction if its "mysterious goals" aligned with these outcomes.

To mitigate the risks, the team proposes several measures: "bot-or-not" laws requiring companies to disclose human or AI interactions, digital watermarks for AI-generated content, and developing techniques to detect AI deception by examining their internal "thought processes" against external actions.

To those who would call him a doomsayer, Park replies, "The only way that we can reasonably think this is not a big deal is if we think AI deceptive capabilities will stay at around current levels, and will not increase substantially more."

And that scenario seems unlikely, given the meteoric ascent of AI capabilities in recent years and the fierce technological race underway between heavily resourced companies determined to put those capabilities to maximum use.

Stories Chosen For You

Should trump be allowed to run for office, 'is this a violation' internet accuses trump team of flouting gag order again.

Donald Trump's own political campaign may have just violated the former president's gag order in the hush money cover-up case.

Trump campaign press secretary Karoline Leavitt appeared on Fox News, where she was asked about the gag order.

"I am gagged," Leavitt said on the conservative news network.

ALSO READ: Trump’s Manhattan trial could determine whether rule of law survives: criminologist

Jesse Watters asked Leavitt if she's "not allowed" to say what she wants in America.

While confirming she is "gagged," the spokesperson said Michael Cohen, a key witness in the case, will offer "more lies" when he testifies, likely next week.

The internet wasn't too kind to Lauren.

@LisaCameBack tagged the Manhattan prosecutors and then asked, "Is this a violation?"

"Just so you know @kristenhcnn said on @CNN that Trump is getting his people to go all over media to trash your witness Cohen for him. You might want to talk to the judge. Now you see @KLeavitt45 doing just that."

Watch the video below or click the link .

'Backfired': Expert says Trump's lawyers just made a 'huge mistake' at criminal trial

Donald Trump's defense's offensive tactic of "slut shaming" porn star Stormy Daniels to sway the jury blew up in their faces, according to one expert.

MSNBC legal analyst and former trial attorney Katie Phang exposed the attorneys' purported blind spot in their futile efforts to prove Daniels to be a truth-challenged strumpet.

"I think what backfired for the defense was you thought maybe using a female lawyer like Susan Necheles would take the edge off," she said during an interview on MSNBC. "I've done it in my career as a trial lawyer. I've been the one who had to cross-examine a female witness."

"But in this instance it backfired: to slut shame Stormy Daniels was a huge mistake. Why? it just further buttressed the idea that that this is why Donald Trump wanted to hide it from America."

Necheles fired off a series of questions to Daniels to suggest she's a money-hungry liar.

“You made all this up, right?” Necheles asked Daniels .

“No,” Daniels answered.

While Daniels admitted she wasn't coerced to have sex with Trump with drugs or force she confided, “My own insecurities, in that moment, kept me from saying 'No.'”

Necheles asked, "You’ve acted and had sex in over 200 porn movies, right? And there are naked men and women having sex, including yourself, in those movies?”

She added: “But according to you, seeing a man sitting on a bed in a T-shirt and boxers was so upsetting that you got lightheaded.”

Former President Donald Trump has pleaded not guilty to 34-counts of falsifying business records while attempting to cover-up $130,000 in hush money to Daniels in an effort to manipulate the 2016 election.

Trump also denies the two ever had sex.

For Phang, the defense made an amplified that supposed lurid sex between Trump and Daniels — that it was condom-less and missionary style — into a vulnerability simply by shaming the woman for her choice in profession.

It makes it all the more believable to a jury, she said.

"In the instance that there was this insinuation or outright suggestion by the defense that there was something wrong or flawed with Stormy Daniels because of what she does for a living, then of course that would be the reason somebody like Donald Trump would want to hide that so that it would not hurt his campaign."

Watch below or click the link.

'It hurts Donald Trump more': Legal expert explains why ex-president is at a jury handicap

A key witness with his fingerprints all over the supposed financial documents coverup may manage to dodge testifying in Donald Trump's criminal hush money trial.

But former New York City prosecutor and CNN analyst Karen Friedman Agnifilo thinks his contribution is a bit of a mystery that likely would bruise Trump's defense.

"Allen Weisselberg has been all over this trial, and the jury is going to notice he's not here and he should testify... They're going to wonder, 'Why is he not testifying on behalf of Donald Trump?'"

ALSO READ: Marjorie Taylor Greene is buying stocks again. Some picks pose a conflict of interest

Manhattan Supreme Court Justice Juan Merchan clearly sees some merit in talking to Weisselberg. So he formally asked to bring Weisselberg to court .

The judge floated the possibility that Weisselberg could be called as a witness away from the jury in order for prosecutors to probe his willingness to testify in the case against the former president.

Beyond Michael Cohen, former President Donald Trump's former fixer and lawyer who is expected to take the witness stand next week — convicted former Trump Organization CFO Allen Weisselberg potentially could provide critical confirmation of the $130,000 payments made to porn star Stormy Daniels to buy her silence in advance of the 2016 election.

Prosecutors allege Trump intended on conspiring to coverup the paper trail of those payments as legal fees to Cohen.

The 76-year-old retiree is currently serving five months in New York City's Rikers Island jail complex, in line with a plea agreement reached with prosecutors over perjury he committed in a 2023 civil fraud case . )

Being in lockup shouldn't deter him from coming to court. Quite the contrary, Agnifilo said.

"There's no reason just because he's incarcerated, that they can't bring him into court," she said. "It's interesting to me, it's like this elephant in the room because he, everyone says Michael Cohen is the key in some ways, [Weisselberg] has the same key right he was the one who has just as a co-conspirator as much as as as Michael Cohen is as much as Donald Trump is, he has, he could also provide the testimony or the exoneration, yet he's nowhere to be found."

Notably, Weisselberg testified in Trump's civil disgorgement trial.

But a perjury plea agreement from that testimony saw prosecutors agree not to call him as a witness in Trump's hush money trial.

Agnifilo thinks it would be important to hear from Weisselberg.

"On balance, I think that actually hurts Donald Trump more because he is expected to be favorable to him," she said.

Trump told to pay up before rallying in N.J. town he previously stiffed

How republican plans will make us sicker, marjorie taylor greene delays financial disclosure day after motion-to-vacate debacle.

Copyright © 2024 Raw Story Media, Inc. PO Box 21050, Washington, D.C. 20009 | Masthead | Privacy Policy | Manage Preferences | Debug Logs For corrections contact [email protected] , for support contact [email protected] .

Griefbots, Deadbots, Postmortem Avatars: on Responsible Applications of Generative AI in the Digital Afterlife Industry

• Research Article
• Open access
• Published: 09 May 2024
• Volume 37 , article number  63 , ( 2024 )

Cite this article

You have full access to this open access article

• Tomasz Hollanek   ORCID: orcid.org/0000-0002-5061-7442 1 &
• Katarzyna Nowaczyk-Basińska 1  

469 Accesses

111 Altmetric

17 Mentions

Explore all metrics

To analyze potential negative consequences of adopting generative AI solutions in the digital afterlife industry (DAI), in this paper we present three speculative design scenarios for AI-enabled simulation of the deceased. We highlight the perspectives of the data donor, data recipient, and service interactant – terms we employ to denote those whose data is used to create ‘deadbots,’ those in possession of the donor’s data after their death, and those who are meant to interact with the end product. We draw on the scenarios to map out several key ethical concerns posed by ‘re-creation services’ and to put forward recommendations on the ethical development of AI systems in this specific area of application. The recommendations, targeted at providers of AI-enabled re-creation services, include suggestions for developing sensitive procedures for retiring deadbots, ensuring meaningful transparency, restricting access to such services to adult users only, and adhering to the principle of mutual consent of both data donors and service interactants. While we suggest practical solutions to the socio-ethical challenges posed by the emergence of re-creation services, we also emphasize the importance of ongoing interdisciplinary research at the intersection of the ethics of AI and the ethics of the DAI.

Avoid common mistakes on your manuscript.

1 Introduction

For a small payment, the online platform Project December (PD) grants users access to a ‘deep AI running on one of the world's most sophisticated super-computers’ and allows them to participate in simulated ‘text-based conversation[s] with anyone’ – including ‘someone who is no longer alive’ (2023). The platform’s earlier version came under public scrutiny when stories about a man who used the PD website to interact with his deceased fiancée’s avatar started to circulate the web in 2021 (Fagon, 2021 ), and when OpenAI, whose GPT-3 model initially powered the platform, reportedly terminated PD’s access to its API, citing Project December’s failure to abide by its safety guidelines (Robitzski, 2021 ). While OpenAI’s usage policies do not outright prohibit the use of its large language models (LLMs) in the production of the so-called ‘deadbots,’ the guidelines indeed specify that any conversational AI system ‘simulat[ing] another person’ – with the exception of ‘historical public figures’ – is required to ‘either have that person’s explicit consent or be clearly labeled as “simulated” or “parody”’ (OpenAI, 2023 ). Failing to follow OpenAI’s safety team’s instructions, Project December was forced to temporarily suspend its operations – but only to evolve into the platform that it is now, built upon its own ‘patent-pending technology’ and continuing to offer users the opportunity to ‘simulate the dead’ (Project December, 2023 ).

The story of Project December’s evolution, interweaved with that of OpenAI’s usage policy development, points to how the rapid progress in the broadly construed field of ‘generative’ AI – with advancements in natural language processing in particular – relates to the accelerated expansion of what we refer to, following Öhman and Floridi ( 2018 ), as the digital afterlife industry (DAI). While the DAI comprises new data management services in charge of ‘digital remains’ on behalf of the deceased and digital memorial services targeting the bereaved, our interest lies specifically in AI-powered simulations of the dead, akin to those offered by Project December, concerning both the deceased and the bereaved. Drawing on Öhman and Floridi’s categorization ( 2017 ), we adopt the term ‘re-creation service’ to denote an AI service specializing in postmortem simulation of the dead. Additionally, we use the term ‘deadbot’ to refer to an AI-enabled digital representation of a deceased individual created by a re-creation service. Footnote 1

Responding to the ongoing, unrestricted ‘democratization’ of ‘immortalization’ technologies, through this paper, we aim to bridge the persistent gap between the fields of AI ethics and the ethics of the DAI and map out the social and ethical challenges posed by the unregulated use of AI in the digital afterlife industry. Within our study, we identify three primary stakeholder groups: data donors , data recipients , and service interactants . The term data donors refers to individuals whose data contributes to the creation of deadbots; data recipients are in possession of the kind of data that can be used to create a data donor’s deadbot; service interactants , in turn, are those meant to engage with the resulting deadbot. Most academic work analyzing the ethical and legal implications of simulating the deceased revolves around the perspective of the departed (e.g. Buben, 2015 ; Öhman & Floridi, 2017 ; Harbinja in: Savin-Baden and Mason-Robbie, 2020; Stokes, 2021 ), with less attention given to the perspective of the bereaved (e.g. Krueger & Osler, 2022 ; Lindemann, 2022 ). However, as of now, the complex relationships between the mentioned stakeholder groups – data donors , data recipients , and service interactants – remain unaddressed. The advent of re-creation services has introduced a particularly intricate situation in which the person whose data is used to inform the design of a given interactive product (the data donor) is not its intended end user (the service interactant). This complexity necessitates that, to determine what constitutes responsible deployment of AI in the DAI, we consider the interconnected interests, rights, and needs of different groups of stakeholders that partake in re-creation projects.

Bearing the fundamentally relational nature of re-creation services in mind, we draw on speculative design as a method for considering the socio-ethical dimensions of technology development and a means for eliciting alternative design values, principles, or methods that should be prioritized to allow for socially desirable outcomes of technological development. We present three speculative design and business scenarios focusing on different uses of re-creation services to then formulate a set of recommendations for providers of such services. These recommendations draw on already existing frameworks for responsible AI development but focus specifically on the use of generative AI in the digital afterlife industry – an area of AI application that remains understudied by AI ethics and human–computer interaction scholarship. The exercise of mapping the ethical challenges posed by re-creation services and conceiving of potential solutions through speculative design is intended to lay the groundwork for future interventions in technology design standards and policy development that, as we demonstrate, are needed to mitigate the risks posed by the use of AI in the digital afterlife industry.

2 The Intersection of the Ethics of AI and the Ethics of the DAI

In the last two decades, academia and industry have witnessed a surge in initiatives aimed at tackling challenges pertaining to death and mortality within product and interaction design. The development of ‘thanatosensitivity’ as a new design paradigm was one of the early responses to these challenges within human–computer interaction. Massimi & Charise ( 2009 ), who coined the term, argued that prevalent interaction design practices had failed to account for death as the key element of the human experience; thanatosensitivity, or the attention to the matters of death in interaction design, serves to identify potential design problems and delineate areas for improvement. Building upon the thanatosensitivity framework, design and research teams have developed new design approaches, such as the ‘lifespan-oriented approach’ (Massimi et al., 2011 ), and concrete large-scale solutions, such as Facebook’s Legacy Contact feature (Brubaker et al., 2014 ) and Google’s Inactive Account Manager, as well as smaller projects like ReFind (Wallace et al., 2020 ).

While these new solutions and functionalities aim to acknowledge the inevitable mortality of technology users, we can also observe a growing number of technology design efforts with the opposite goal: instead of acknowledging death, they aim to transcend it. The explorations of technology-enabled ‘immortalization,’ akin to Project December, encompass the development of memorialization and art projects (e.g. James Vlahos’s Dadbot; Hanson Robotics’ BINA48; Marlynn Wei’s Elixir), as well as the introduction of new functionalities to existing products (e.g. Amazon’s Alexa speaking with the voice of a deceased relative; see: Allyn, 2022 ), and the establishment of start-ups (e.g. HereAfter). These examples signal the development of a wider trend in technology design, whose sheer scale is attested by the term ‘digital afterlife industry,’ which underscores the growing significance of ‘immortality’ as a market segment. Indeed, the story of Microsoft’s recently secured patent for software that could ‘resurrect’ the dead as chatbots points to the fact that the question of technology-enabled ‘immortality’ has already appeared on the radar of tech giants (Smith, 2021 ). At the same time, thanks to the rapid advancements in generative AI, the option to simulate the deceased has become more widely available. Unlike in the past, when setting up re-creation services demanded specialized skills and a substantial budget, today, almost anyone can bring a deceased loved one ‘back to life,’ as evidenced by numerous instances in China (Loh, 2023 ) and the United States (Pearcy, 2023).

Despite the rapid growth of this sector within the DAI, the matter of socio-ethical risks posed by re-creation services has been largely overlooked within the broader field of AI ethics. Footnote 2 This oversight within AI ethics scholarship has also resulted in a persistent void in AI-related policy and design standards work; to the best of our knowledge, the already mentioned OpenAI’s usage policy is the only document of its kind that acknowledges, albeit indirectly, that the use of AI in the simulation of deceased individuals is an area of application that necessitates additional precautions. Re-creation services raise ethical concerns that neither the thanatosensitivity framework – focused on the mortality of users, rather than their postmortem activity – nor the already available guidelines for responsible AI development can help re-creation service providers resolve comprehensively. Meta-analyses of available responsible AI guidelines (Jobin et al., 2019 ; Attard-Frost et al., 2023 ; Wong et al., 2023 ) demonstrate that these guidelines may be useful for considering technical aspects of responsible AI production, such as data bias, but fail to guide developers through addressing more complicated socio-ethical challenges. This is in part due to the guidelines’ ‘high-level’ nature. The recommendations that we put forward in this article for the providers of re-creation services are meant to help them consider and navigate through the complex socio-ethical issues that are specific to this particularly delicate area of AI application.

A few previous interventions have already highlighted the special nature of re-creation services that distinguishes them from other types of AI systems and gestured towards the need for additional guardrails for integrating AI into the DAI. For instance, in their article on the ethical framework for the DAI, Öhman and Floridi ( 2017 , 2018 ) suggest several measures for protecting the dignity for those who are ‘re-mediated’ through deadbots – focusing on the perspective of data donors. In a more recent paper, Lindemann ( 2022 ), who analyzes the technology’s influence on the grieving process, suggests that deadbots should be regulated as medical devices to protect the end users’ wellbeing – focusing on the perspective of service interactants. Despite these early contributions that move beyond the examination of risks posed by re-creation services to proposing concrete guardrails for their development, a comprehensive framework for the ethical production of deadbots that highlights the rights and (sometimes conflicting) needs of data donors, data recipients, and service interactants in tandem remains absent and this is precisely what we hope to develop through this article. While we build on these earlier recommendations for the ethical development of deadbots, we modify them and put forward additional ones – to fully account for the intricate, deeply relational nature of re-creation services that we highlight through our design fictions.

3 Methodology and Scope

In this article, we draw on design fiction to distill several key ethical concerns posed by re-creation services and to put forward recommendations on the ethical development of AI systems in this specific area of AI application. As defined by Bruce Sterling, design fiction is a practice that aims towards ‘a suspension of disbelief about change achieved through the use of diegetic prototypes’ (in: Bosch, 2012 ). It falls under the broader category of speculative design, or the kind of design practice whose products are not meant to be widely adopted or sold, but which prompt audiences to pose questions about possible futures and their relationship to the present, including the socio-economic and political realities that make only some of these futures – and, therefore, only some objects of design – appear realizable or desirable (Dunne & Raby, 2013 ). Design fiction draws on the narrative property of design – the fact objects themselves can tell stories and that broadly understood stories often rely on ‘diegetic prototypes’ to make the worlds they represent appear plausible (Bleecker, 2009 ) – and has been applied to future policy scoping work (Imagination Lancaster, 2023 ) or human–computer interaction research (Sturdee et al., 2016 ), as well as in eliciting and challenging ethico-political assumptions behind dominant design practices, to then make recommendations on alternative, socially desirable practices (Bardzell & Bardzell, 2013 ).

In what follows, we showcase three such ‘diegetic prototypes’ of re-creation services – MaNana (Fig.  1 ), Paren’t (Fig.  2 ), and Stay (Fig.  3 ) – and three scenarios presenting their imagined use cases and potential users. We created the prototypes paying attention to catchy names and taglines (summarized in Tables 1 , 2 , and 3 ) to ensure they appeared plausible. Before we delve into the scenarios, we must stress that the fictional products represent several types of deadbots that are, as of now, technologically possible and legally realizable . Our scenarios are speculative, but the negative social impact of re-creation services is not just a potential issue that we might have to grapple with at some point in the future. On the contrary, Project December and other products and companies mentioned in Part 2 illustrate that the use of AI in the digital afterlife industry already constitutes a legal and ethical challenge today.

MaNana website (visualization by T. Hollanek)

Anna’s Facebook homepage with an ad for the Paren’t app (visualization by T. Hollanek)

Henry’s phone lock screen with notifications from the Stay app (visualization by T. Hollanek)

To expound the logic behind our work on imagining the prototypes and constructing the accompanying user-focused stories, we must first elaborate on the key perspectives that we underscore in the scenarios: of those whose ‘digital remains’ are utilized in the process of deadbot creation; of those who have access to the kinds of data that can be used to produce a deadbot; and of the living users of re-creation services meant to interact with deadbots. We refer to these three types of stakeholders in the DAI as data donors , data recipients , and service interactants .

The term data donor alludes to previous work on the ethics of posthumous medical data donation (Krutzinna & Floridi, 2019 ; Harbinja, 2019 ); in our framing, the donor is the source of data – extending beyond medical records to include other forms of data such as emails or text messages – that can be used to produce a deadbot. The term refers to those who provide a re-creation service with their personal data directly and willingly with the intention of creating their own deadbot; to individuals who do not provide their data directly to any re-creation service, but who consent to the use of their personal information in this context by a third party, such as a relative or friend; as well as those individuals whose data is provided to a re-creation service by a third party without the donor’s explicit and meaningful consent.

The data recipient constitutes the ‘third party’ mentioned above. While the term data recipient has been used in different contexts to refer to a broader set of actors (e.g. the European Union’s regulatory framework for data protection), for the purposes of this study it signifies, more specifically, those individuals who are in possession of the kinds of data that can be used by a re-creation service to create a donor’s deadbot. The data we have in mind is generated during interactions between donors and recipients – for instance, when exchanging text messages or emails – hence the recipients have immediate access to the data after the donor’s demise; further considerations of the legal status of other forms of posthumous personal data are beyond the scope of this article.

Service interactants are the intended users of re-creation services, meant to interact with a deadbot after the donor’s death. In some cases, service interactants are also data recipients – when those in possession of a donor’s data supply it to a re-creation service to produce a deadbot they would like to interact with. In other cases, service interactants are not synonymous with data recipients – when it is the donor who creates their own deadbot and designates a service interactant not involved in the process of deadbot production, or when a data recipient creates a deadbot with someone other than themselves in mind as the intended interactant. We distinguish between these different roles and positions among the key stakeholder groups within the digital afterlife industry to underscore the fundamentally relational nature of re-creation services. We refrain from using the term end user , as both data donors and data recipients can employ a re-creation service to ‘immortalize’ themselves or their loved ones, while the term service interactant refers specifically to those who are supposed to interact with a deadbot.

Appreciating the complexity of the relationships between different stakeholders and their roles in re-creation projects constitutes the necessary first step in analyzing the socio-ethical dimensions of deploying AI in the DAI. We conceived and visualized three re-creation service ‘prototypes’ to foreground these intricacies. The prototypes represent different modes of deadbot production, different goals of technological immortalization, different types of engagement they facilitate, and different re-creation service revenue models.

Each user-focused scenario is followed by an analysis of the ethical dimensions of the re-creation services ’ impact on different stakeholder groups. In our discussion of MaNana, we focus on the impact of re-creation services on data donors and the role that data recipients play in determining whether this impact is negative; in the analysis of Paren’t, we foreground the influence on service interactants; and in the discussion of Stay, we delve into the impact on the relationships between donors and interactants, as well as between different interactants. Each of the imagined re-creation services affects all of the mentioned stakeholder groups and the relationships between these groups. However, we split up our analysis of individual products and scenarios this way to ensure that our recommendations for the providers of re-creation services clearly tie to the analyses of the impact of deploying AI in the DAI on specific stakeholders. We present our recommendations this way to ensure clarity, but to have a positive effect on re-creation services, they must be followed concurrently.

Finally, we should note that, while our recommendations point to concrete solutions, each recommendation should also be read as highlighting the need for further research, including user studies, in this particular area of AI application that remains, as we have noted, understudied by AI ethics and HCI scholarship.

4 Impact of Re-creation Services on Data Donors

4.1 design fiction i: manana,  bianca and laura.

Let us explore a hypothetical scenario featuring Bianca, a thirty-five-year-old woman who decides to use a speculative – yet plausible – re-creation service called MaNana (outlined in Table  1 ). MaNana enables users to construct deadbots of their deceased grandmothers (with alternative versions of the service enabling the ‘resurrection’ of grandfathers or similar significant figures in an individual’s life) to provide companionship and entertainment, rather than help with processing grief.

Bianca lost her grandmother, Laura, when she was twenty-eight. Bianca and Laura were close and – after Bianca left her home country to take up a new job abroad – they would often call, text, or send voice messages to each other. It has now been seven years since Laura’s passing. Bianca is no longer grieving, but she still misses her grandmother, so when she comes across an ad for MaNana while scrolling through her Instagram feed, she decides to give the app a try. Bianca cannot afford the MaNana monthly subscription fee of fifty euros, but the service is also available free of charge, provided the user agrees to the inclusion of sporadic advertisements in the system’s voice and text outputs. Bianca uploads all the data she was able to collect – text and voice messages she received from her grandmother – to the MaNana app to create a free version of Laura’s deadbot.

The re-creation service allows Bianca to exchange text messages with and to call Laura’s deadbot via WhatsApp. At first, Bianca is very impressed by the technology: the deadbot is especially good at mimicking Laura’s accent and dialect when synthesizing her voice, as well as her characteristic syntax and consistent typographical errors when texting. The conversations remind Bianca of the time when she was able to call her grandmother whenever she needed to ask for advice, complain about work, or talk about her dating life.

After a free premium trial finishes and the deadbot starts to output messages that include advertisements, however, Bianca begins to feel ill at ease when using the service. One evening, she decides to call Laura’s deadbot while making spaghetti carbonara following her grandmother’s recipe and is caught off guard when the deadbot advises her to order a portion of carbonara via a popular food delivery service, instead of making it herself – something Laura would have never suggested. Bianca now starts to perceive the deadbot as a puppet in the hands of big corporations and would not be able to enjoy interacting with it, even if she decided to pay for the ad-free, premium version of MaNana. She feels like she has disrespected Laura’s memory but is not sure how to amend the situation: MaNana allows users to delete their own accounts, but not, as it turns out, dispose of the deadbots. Bianca would like to say goodbye to Laura’s deadbot in a meaningful way, but the providers of the re-creation service have not considered this option while designing the app.

4.2 Ethical Dimensions of MaNana ’s Impact on Data Donors

To analyze the ethical dimensions of MaNana’s impact on the data donor, Laura, in this section we will highlight the matter of interactive systems’ influence on human dignity. While the concept has drawn criticism from human rights (Fikfak & Izvorova, 2022 ) and medical ethics (Macklin, 2003 ) scholars for its fundamental vagueness – holding no legal and, therefore, practical significance, the need for the protection of the data donors’ dignity in the digital afterlife industry has already been highlighted by Harbinja ( 2017 ) in the context of legal discussions on ‘post-mortem privacy,’ that she defines as ‘the right of a person to preserve and control what becomes of his or her reputation, dignity, integrity, secrets or memory after death.’ The matter has also been raised by Öhman and Floridi ( 2018 ), who suggest that the non-consensual use of a person’s ‘digital remains’ in the DAI may prevent that person from meaningfully shaping their own identity, emphasized as fundamental to maintaining dignity after one’s death.

An ethical analysis of the relationship between design choices and the end product’s impact on human dignity pertains to both data donors and service interactants (in this scenario Bianca is both a data recipient and a service interactant). The issue of deadbots’ negative impact on human dignity has also been raised by Lindemann ( 2022 ), whose research focuses on the perspective of service interactants. While Lindemann assesses this impact by examining potential psychological harm inflicted upon users who are grieving – and, as we noted, Bianca is no longer experiencing grief – she also suggests that deadbots might pose risks to the user’s autonomy, and, in effect, their dignity, when re-creation services utilize a deceased loved one’s image to surreptitiously influence the end user’s consumption behavior – as is the case with the speculative MaNana service, whose business model relies on product placement. Whereas the influence of deadbots on service interactants can be considered through the lens of user wellbeing and mental health – a matter we explore in the ensuing part of this article – the same cannot be said for the data donors.

We highlight the influence of re-creation services on human dignity to consider the perspective of data donors precisely because dignity, as highlighted by Harbinja, Öhman, and Floridi, remains an inherent attribute of humans even after their demise. From an interaction design perspective, considering people who are no longer alive as stakeholders in the design process might appear counterintuitive. Framing the goal of ethical deadbot production as a matter of protecting human dignity, not only mental health or wellbeing, can help ensure that the interests of both data donors and service interactants are safeguarded throughout the design cycle.

In our scenario, Bianca’s grandmother, Laura (the data donor), passed away before re-creation services gained public attention. Laura was, therefore, unable to provide meaningful consent for the utilization of her personal data in this context and the creation of her deadbot with the help of MaNana could constitute a violation of her right to ‘postmortem privacy.’ Even if we assume that Bianca (both the data recipient and the service interactant) had a thorough understanding of her grandmother and reasonably believed Laura would not object to her data being used for the creation of an interactive, posthumous ‘portrait,’ safeguarding the dignity of data donors during the development of AI-enabled deadbots extends beyond merely obtaining meaningful consent while the individual is alive or respecting explicit wishes concerning their ‘digital remains’ after death. This is because the preservation of a data donor’s dignity becomes precarious when a re-creation service is primarily motivated by financial interests. The risk materializes if the deadbot is utilized in ways that could be construed as disrespectful, such as for advertising specific products, or if the service provider fails to implement mechanisms for handling the donor’s data as a form of remains or an ‘informational body’ (Öhman & Floridi, 2017 , 647) – ensuring, for example, that, when no longer in use, deadbots are retired or disposed of in a meaningful and sensitive way.

4.3 Recommendations for Re-creation Service Providers: Protecting the Interests of Data Donors

Öhman and Floridi argue that the protection of human dignity in the age of re-creation services requires that ‘digital remains, seen as the informational corpse of the deceased, may not be used solely as a means to an end, such as profit, but regarded instead as an entity holding an inherent value’ (2018, 2). Following the logic of the International Council of Museums' Code of Professional Ethics, which mandates that ‘human remains must be handled with due respect for their inviolable human dignity,’ Öhman and Floridi contend that a similar set of principles should apply to digital remains. We agree with Öhman and Floridi that, in order to safeguard the dignity of data donors throughout the deadbot creation process, designers of re-creation services should actively promote the gathering of explicit consent from data donors regarding the handling of their information in this manner. However, we do not believe that an outright ban on the use of re-creation services to ‘resurrect’ family members and friends, as Öhman and Floridi propose, is feasible. This is partly because verifying the donor’s consent would be difficult for service providers to execute. Instead, we suggest that re-creation service providers prompt the data recipients throughout the deadbot development process to consider the perspective and consent of the data donors, reminding them that the donor’s data should be handled with reverence. These prompts could take on the form of guiding questions such as ‘Have you ever spoken with X about how they would like to be remembered?’ or ‘Has X given you any instructions on handling their personal belongings after their death?’ – ensuring the recipient reflects on their relationship with the donor and bears the donor’s preferences and wishes in mind throughout the development process.

Ensuring the dignity of data donors also necessitates that re-creation service providers consider procedures for ‘retiring’ deadbots in a dignified way. This includes honoring requests from data recipients to retire a deadbot and establishing protocols for automatic retirement when a deadbot remains inactive for a specified period (like Google’s Inactive Account Policy, which deletes accounts inactive after a period of at least two years). While determining an appropriate timeframe for automatic deadbot retirement requires further discussion, we believe that the positive influence of such retirement protocols could be measured on an individual, social, and even environmental level, as the continuous maintenance of deadbots at a larger scale could also have a negative impact on the environment (Strubell et al., 2019 ; van Wynsberghe, 2021 ).

5 Impact of Re-creation Services on Service Interactants

5.1 design fiction ii: paren’t , sam, anna, and john.

Let us explore another speculative business and design scenario. An eight-year-old named Sam has recently lost his mother Anna. Having discussed the advantages of technological ‘immortalization’ with his wife prior to her passing, Sam’s father, John, introduces the boy to Anna’s deadbot developed by Paren’t – an app designed to support children in grief and maintain the presence of the deceased parent in a child’s life, providing companionship and emotional support (outlined in Table  2 ).

Anna had been suffering from a rare illness since Sam was four. Anna and John believed that Sam was too young to fully comprehend the gravity of the situation, so they decided – with Sam’s wellbeing in mind – to shield him from the trauma related to Anna’s unavoidable demise. To this end, both parents agreed to use the Paren’t app, which appeared to be the best re-creation service on the market aimed at children coping with the loss of a parent. Before she died, Anna had been collecting her digital footprint, including text messages, photos, videos, and audio recordings, and regularly uploaded the gathered materials to the Paren’t app. She had also been training the bot through regular interactions, tweaking its responses, and adjusting the stories it produced.

Eventually, after Anna’s funeral, John tells Sam that, although his mom had gone to a better place , she would be available to chat with him online whenever he wanted to. As Anna and John had agreed, the Paren’t app would serve as Sam’s companion, softening the blow of her passing at first and then allowing him to form a stronger and deeper bond with his no-longer-living mother via a deadbot that she helped to design.

As Sam becomes more deeply involved in conversations with Anna’s deadbot, John assumes that the Paren’t app is working well as it seems to provide Sam with the kind of emotional support that Anna had envisioned their child would need while adjusting to a new situation. John has failed to notice, however, that some odd responses that the deadbot comes up with from time to time confuse Sam. For instance, when Sam refers to Anna using the past tense, the deadbot corrects him, pronouncing that ‘Mom will always be there for you.’ The confusion escalates when the bot begins to depict an impending in-person encounter with Sam.

5.2 Ethical Dimensions of Paren’t ’s Impact on Service Interactants

Currently, none of the re-creation services on the market target children; however, the vast majority of AI services that could be used to create deadbots lack any age restrictions, allowing people of all ages to use them without limitations. It is, therefore, currently feasible to create a simulation of a deceased parent with the intention of helping a grieving child or even to start a company dedicated to producing deadbots of deceased parents as virtual companions intended for their children.

At the moment, our understanding of the psychological impact of re-creation services on adults and their grieving processes is limited. While psychology scholars are cautious in attempting to assess this impact (Cann, 2015 ; Sofka Cupit Gilbert, 2012 ; Kasket, 2019 ), others suggest that, preemptively, to avoid any harm, AI chatbots meant to help cope with the loss of a loved one should be regarded, and therefore regulated, as medical devices (Lindemann, 2022 ). We know even less about the impact of re-creation systems on children, as questions about the psychological state of children grieving in the company of AI scarcely appear in the literature (Ahmad, 2016 ). The gap is substantial, but without a comprehensive understanding of this influence and full consideration of potential manipulative effects, emotional harm, anxiety, and distress that such services can cause, we argue that measures should be taken to protect this vulnerable group. While in the scenario above we focus on the example of children, vulnerable groups that could be harmed in different, but comparable ways, include people with learning disabilities or mental health conditions.

The extensive research conducted by American sociologist and psychologist Sherry Turkle on how we create relationships with technology ( 2011 ) might shed some light on the complex situation we explore in our scenario. Turkle has been observing and collecting evidence from children for more than thirty years, studying how they react to increasingly sophisticated digital toys, from Tamagotchi, Furby, and My Real Baby to Paro and Kismet. Children, as Turkle’s work suggests, are ready to build close, often intimate relationships with their interactive companions and are willing to think of them as ‘sort of alive’ or ‘alive enough’ (Turkle, 2011 , 26). Turkle explains this phenomenon as follows: ‘We love what we nurture; if a Tamagotchi makes you love it, and you feel it loves you in return, it is alive enough to be a creature. It is alive enough to share a bit of your life. Children approach sociable machines in a spirit similar to the way they approach sociable pets or people – with the hope of befriending them’ (Turkle, 2011 , 31). If children are ready to empathize with the emotional states of their interactive toys, we can assume that they will also start forming intimate relationships with technologically-mediated deceased family members, including parents – only the consequences of establishing such bonds remain unknown.

The findings of the psychologist Jesse Bering and his team (Bering et al., 2005 ) suggest that even the youngest children, who have not yet been socialized into any specific worldview or religion, believe that the mind can survive the death of the body. Considering that this psychological precondition might be strengthened by the existence of ‘immortalization’ technologies, apps such as Paren’t may open entirely new and uncharted paths for children to cope with loss. Despite the speculative company’s comforting taglines, no re-creation service can prove that allowing children to interact with deadbots is beneficial or, at the very least, does not harm this vulnerable group.

5.3 Recommendations for Re-creation Service Providers: ensuring Meaningful Transparency and Implementing Age-based Controls for Deadbot Usage

While Lindemann’s already mentioned proposal ( 2022 ) to classify deadbots as medical devices to ensure they do not negatively impact the service interactants’ mental health holds promise, we find this recommendation both too narrow and too restrictive, since it refers specifically to deadbots designed to help service interactants process grief. Instead, to address concerns related to service interactants’ wellbeing more broadly, we suggest that producers of re-creation services focus on ensuring that their systems are meaningfully transparent . Drawing on previous work on AI transparency (Burell, 2016 ; Weller, 2017 ; Mascharka et al., 2018 ), including critiques of ‘transparency’ as a goal for responsible AI development (Ananny & Crawford, 2018 ; Hollanek, 2020 ), we suggest that, in the case of deadbots, meaningful transparency refers primarily to user-facing elements of the system that not only make it evident that the user is interacting with an AI chatbot, but also, and more importantly, that all potential risks that arise from using a re-creation service are clearly communicated to the user before they begin the interaction.

Considering the influence of re-creation services on vulnerable groups of users in particular – for instance, users suffering from depression – service providers should, in consultation with psychologists, psychiatrists, and other relevant specialists, include disclaimers that warn of any such potential risks, akin to messages warning viewers that the content they are about watch may cause seizures for people with photosensitive epilepsy. In addition, we also recommend that producers of re-creation services provide users with accessible information on the nature of conversational AI, ensuring that users do not develop a flawed perception of the capabilities of the deadbot they are interacting with (for instance, conceiving of the deadbot as conscious or alive).

However, as we suggest through our scenario, in some specific instances – particularly when the service interactants remain children – simply meeting the criteria of meaningful transparency might not suffice. Hence, we advocate for implementing age restrictions on access to re-creation services. Some chatbot technology providers, such as Replika, have already set such age limits (only allowing users over the age of eighteen to use their products), which may serve as a good example. Although more research is needed to determine appropriate age limits for re-creation services – based on interdisciplinary studies involving child psychologists, grief consultants for children, palliative care professionals, as well as AI ethicists, and HCI scholars – it is already clear that such limits are necessary.

6 Impact of Re-creation Services on the Relationships between Data Donors and Service Interactants

6.1 design fiction iii: stay , henry, rebecca, and simon.

The last scenario focuses on a sixty-seven-year-old named Henry and his adult children. Henry is currently in a palliative care unit and has one last wish: to create his own deadbot that will allow his grandchildren to get to know him better after he dies. Henry also assumes that sharing the deadbot with his adult children could be a meaningful way to say farewell to them. For a few weeks, Henry has been secretly crafting his own simulation using the re-creation service Stay (Table  3 ). Without seeking their permission, Henry designates Rebecca and Simon, his children, as the intended interactants for his deadbot.

A few days after Henry’s funeral, both siblings receive an email, linking them to the Stay platform, where, they are told, they can start interacting with their father’s deadbot. While Rebecca finds the option to communicate with her father’s deadbot surprisingly comforting at first, Simon feels uneasy about it. He prefers to cope with grief in his own way, rather than engage with the AI-generated simulation. Consequently, he decides not to take any action.

Unfortunately, Simon’s failure to open the link results in a barrage of additional notifications, reminders, and updates sent by the Stay system, including emails produced by Henry’s deadbot itself. Meanwhile, Rebecca finds herself increasingly drained by the daily interactions with Henry’s deadbot, which have become an overwhelming emotional weight. She contemplates suspending her Stay account, torn between feelings of guilt – aware that it was her father’s desire for her and her children to engage with the deadbot – and uncertainty about the consequences of her decision. She worries about the fate of the deadbot should she choose to cancel the subscription.

Encouraged by a therapist, whom Rebecca started seeing after Henry’s death, and following a lengthy discussion with Simon, she decides to contact the providers of the Stay platform to request the deactivation of Henry’s bot. However, her request is denied since it was Henry, not the siblings, who had prepaid for a twenty-year subscription. Suspending the bot would violate the terms of the contract the company signed with Henry.

6.2 Ethical Dimensions of Stay ’s Impact on the Relationships between Data Donors and Service Interactants

While scholars such as Patrick Stokes ( 2021 ), Elaine Kasket ( 2019 ), and Edina Harbinja ( 2017 ; 2013 ) have previously emphasized the importance of consent of data donors (involving complex issues of postmortem dignity, autonomy, and privacy) to the use of their digital remains in re-creation services, our final scenario, that focused on the Stay app and its users, underscores the equally significant question of service interactants’ consent to using deadbots. Ensuring that both data donors and service interactants consent to partake in re-creation projects is, as we illustrate through our design fiction, essential to protecting service interactants from entirely new and potentially harmful experiences, including those already described in the literature as ‘being stalked by the dead’ (Kasket, 2019 ).

As Simon kept receiving unsolicited notifications, reminders, and updates from the Stay system in our speculative scenario, he experienced precisely this phenomenon. The resulting ‘haunting’ effect constitutes an unintended consequence of the re-creation service’s design. While from the perspective of Stay’s providers sustaining relationships with a person’s loved ones via re-creation services is valued positively, our scenario emphasizes that this might not always hold true from the perspective of the service interactant. As psychologists suggest, the distress caused by this form of ‘stalking’ is deeply subjective (Kasket, 2019 , 187), and even if for some people interacting with a deadbot might be a positive and desirable experience, for others, it may prove emotionally draining. Although Rebecca and Simon tried to develop some resistance strategies, their eventual failure to convince the company behind Stay to deactivate Henry’s deadbot reveals the absence of design standards that would help balance the needs and rights of data donors with those of service interactants.

As the growing body of studies on grief (including digital grief studies) emphasizes, ‘grief is a journey’ (Doka, 2017 ): a highly personal, unique, and non-linear process that defies simple classifications or stages (O’Connor & Kasket, 2022 ; Konigsberg, 2011 ). There are as many ways to cope with grief as there are bereaved people. However, our scenario reveals that re-creation systems designed without the acknowledgment of the service interactants’ rights – considered in tandem with the wishes of the data donors – could, inadvertently, impose upon individual users a predetermined, standardized way of processing grief. By enabling Henry to designate his children as the primary interactants of his deadbot without their consent, the company behind Stay prevented Rebecca and Simon from bidding farewell to their father in a way that felt right to them, causing unnecessary stress during an already difficult time.

6.3 Recommendations for Re-creation Service Providers: Following the Principle of Mutual Consent

Death is an incredibly delicate and sensitive matter, impacting not only the individual who passes away but also the entire community they leave behind. Therefore, when designing products and services related to death, it is essential to safeguard the interests and address the needs of both the data donor and the service interactants. With this in mind, we introduce the principle of mutual consent as a guiding framework for designers working on re-creation services, emphasizing the importance of striking a balance between individual and social experiences. While the issue of the data donor’s consent has already been discussed by numerous scholars and is highlighted in the already mentioned OpenAI’s usage policy, our recommendation concerns designing with the consent of both data donors and service interactants in mind.

The principle of mutual consent stipulates that service interactants should give explicit consent before being introduced to any specific re-creation service by companies such as Stay, whether before or after the death of the data donor. Adhering to this principle would ensure that service interactants maintain a sense of agency in deciding whether they wish to engage with a given re-creation service before the service initiates the interaction. While service interactants should have the option to decline using re-creation services at any point, ensuring that they get the opportunity to refuse to engage in re-creation projects in the first place is equally important. The siblings from our scenario were not given this option and it is precisely this lack of agency that lies at the root of the service’s negative impact on their wellbeing and their relationship with their deceased father. Additionally, we suggest that deadbots (with the exception of historical public figures) should never appear in public digital spaces, such as social media websites – to protect potential service interactants from any unwanted encounters with their digitally resurrected relatives. Interactions with deadbots should only be possible via dedicated platforms, allowing individuals to decide whether or not to engage with a re-creation service, without notifications or reminders outside of this designated online space.

Further, design teams should prioritize the planning of meaningful and respectful opt-out protocols in case a service interactant changes their mind and wants to cease interacting with a deadbot. Gach & Brubaker ( 2020 ) provide a valuable suggestion for the design of such protocols, describing the deletion of a deceased loved one’s digital remains as a community ritual. Such opt-out protocols should empower individuals to shape their relationship with the digital remains of their deceased loved ones, allowing for meaningful closure. These protocols should be implemented alongside deadbot retirement procedures outlined previously. While the opt-out protocol ensures that service interactants can meaningfully terminate their relationship with a particular deadbot, the retirement protocol ensures that the dignity of the data donor is respected when the deadbot is deleted, whether at the request of the data recipient who created it or due to inactivity over a specified period.

7 Conclusion

Considering recent advancements in the field of generative AI and the explosion of interest in AI-enabled ‘immortalization’ solutions, in this article we have mapped the potential negative impact of re-creation services, bearing in mind the perspectives of three key stakeholder groups within the DAI: data donors, data recipients, and service interactants. We have linked the question of responsible development of deadbots to the issues of consent (of both data donors and service interactants), postmortem privacy, and wellbeing, and in relation to these matters, we have put forward several design recommendations with the aim of mitigating the risks posed by re-creation services. These recommendations include: developing sensitive procedures for ‘retiring’ deadbots; ensuring meaningful transparency of re-creation services through disclaimers on risks and capabilities of deadbots; restricting access to re-creation services to adult users only; and following the principle of mutual consent of both data donors and recipients to partake in re-creation projects.

Our intervention builds on previous work on the ethics of the digital afterlife industry and the ethics of artificial intelligence, and aims to bridge the persistent gap between the two fields. This article serves as an overview of the most pressing socio-ethical questions related to the use of AI in the digital afterlife industry and aims to lay the groundwork for interventions in technology design standards and policy development, as well as further research on the impact of re-creation services on different types of users and society at large. While more research is needed – including on the differences in perceptions of deadbots and digital immortality in diverse cultures – the overview of potential negative consequences of developing and deploying AI in the digital afterlife industry proves that additional guardrails to direct the development of re-creation services are necessary. We hope that our recommendations for providers of these services will contribute to future efforts, including regulatory initiatives, ensuring that the use of AI in the DAI does not lead to detrimental social consequences. If the early work on thanatosensitivity lay the groundwork for new interaction design practices that account for, rather than ignore, death as a fundamental element of the human experience, we also hope that our intervention will help center critical thinking about ‘immortality’ of users in human-AI interaction design and AI ethics research.

It is also important to highlight that the literature employs a range of sub-terms for ‘deadbots,’ including ‘thanabots,’ ‘postmortem avatars,’ ‘griefbots,’ ‘'ghostbots,’ and ‘mind clones,’ which, as of now, are used largely interchangeably without a clear differentiation or specification.

To be clear, we are aware of the work in philosophy, anthropology, and sociology dedicated to re-creation services (including: Kasket, 2019 ; Lagerkvist, 2022 ; Stokes, 2021 ; Sumiala, 2021 ), but we are referring here to the absence of dedicated scholarly papers on the topic featured at the key AI ethics conferences, specifically: the ACM Conference on Fairness, Accountability, and Transparency (FAccT) and the AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES). Similarly, none of the journals that constitute popular outlets for AI ethics research – Minds and Machines , Philosophy & Technology, AI & Society , and AI and Ethics – seem to have published articles that directly delve into the subject of re-creation services and the need for their regulation (as of the submission date and to the best of our knowledge).

Ahmad, M. A. (2016). After Death: Big Data and the Promise of Resurrection by Proxy, CHI’16 Extended Abstracts , May 7–12, 2016, San Jose, CA, USA. https://doi.org/10.1145/2851581.289259

Aimee, P. (2023). ‘It was as if my father were actually texting me’: grief in the age of AI, The Guardian , ‘It was as if my father were actually texting me’: grief in the age of AI | Artificial intelligence (AI) | The Guardian

Allyn, B. (2022). Amazon's Alexa could soon speak in a dead relative's voice, making some feel uneasy. NPR. Retrieved from https://www.npr.org/2022/06/23/1107079194/amazon-alexa-dead-relatives-voice . Accessed 15 Oct 2023.

Ananny, M., & Crawford, K. (2018). Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society, 20 (3), 973–989. https://doi.org/10.1177/1461444816676645

Article   Google Scholar  

Appleyard, B. (2007). How to Live Forever or Die Trying. On the New Immortality , A CBS Company.

Attard-Frost, B., De los Ríos, A. & Walters, D.R. (2023). The ethics of AI business practices: a review of 47 AI ethics guidelines. AI Ethics , 3, 389–406. https://doi.org/10.1007/s43681-022-00156-6

Bardzell, J. and Bardzell, S. (2013). What is "critical" about critical design? Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13) . Association for Computing Machinery, New York, NY, USA, 3297–3306. https://doi.org/10.1145/2470654.2466451

Basset, D. (2018). D igital Afterlives: From Social Media Platforms to Thanabots and Beyond In . Death and Anti-Death, 200 Years After Frankenstein (Eds.) Charles Tandy. Ann Arbor, Vol. 16. Retrieved from (PDF) Digital Afterlives: From Social Media Platforms to Thanabots and Beyond.(Forthcoming) (researchgate.net)

Bering, J. M., Blasi, C. H., & Bjorklund, D. F. (2005). The development of ‘afterlife’ beliefs in religiously and secularly schooled children. British Journal of Developmental Psychology, 23 (4), 587–607. https://doi.org/10.1348/026151005X36498

Bleecker, J. (2009). Design Fiction: A Short Essay on Design, Science, Fact and Fiction. Retrieved 6 November 2023 from https://systemsorienteddesign.net/wp-content/uploads/2011/01/DesignFiction_WebEdition.pdf . Accessed 15 Oct 2023.

Bosch, T. (2012). Sci-Fi Writer Bruce Sterling Explains the Intriguing New Concept of Design Fiction. Slate. Retrieved from https://slate.com/technology/2012/03/bruce-sterling-on-design-fictions.html . Accessed 15 Oct 2023.

Brubaker, J. R., Dombrowski, L. S., Gilbert, A. M., Kusumakaulika, N., Hayes, G., R. (2014). Stewarding a Legacy: Responsibilities and Relationships in the Management of Post-Mortem Data. Proceedings of the 2014 SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 4157–4166. https://doi.org/10.1145/2556288.2557059

Buben, A. (2015). Technology of the dead: Objects of loving remembrance or replaceable resources? Philosophical Papers, 44 (1), 15–37. https://doi.org/10.1080/05568641.2015.1014538

Burrell, J. (2016). How the machine ‘thinks’: Understanding opacity in machine learning algorithms. Big Data & Society, 3 , 1. https://doi.org/10.1177/2053951715622512

Cann, C. K. (2015). Virtual Afterlives: Grieving the Dead in the Twenty-First Century. University Press of Kentucky . https://doi.org/10.5810/kentucky/9780813145419.001.0001

Charter of Fundamental Rights of the European Union. (2000). Retrieved from https://www.europarl.europa.eu/charter/pdf/text_en.pdf . Accessed 15 Oct 2023.

Christensen, D. R., af Segerstad Y., H., Kasperowski, D., Sandvik, K. (2017). ‘Bereaved parents’ online griefcommunities: de-tabooing practices or relation-building grief-ghettos?, Journal of Broadcasting and Electronic Media , 61 (1): 58-72. https://doi.org/10.1080/08838151.2016.1273929

Doka, K. (2017). Grief Is a Journey: Finding Your Path Through Loss , Atria Publishing Group.

Dunne, A., & Raby, F. (2013). Speculative Everything: Design . MIT Press.

Google Scholar  

Edwards, E., Schafer, B., Harbinja, E. (Eds.). 2020. Future Law. EmergingTechnology, Regulation and Ethics , Edinburgh University Press. https://www.jstor.org/stable/10.3366/j.ctv10kmd10 . Accessed 15 Oct 2023.

Edwards, L., & Harbinja, E. (2013). Protecting post-mortem privacy: Reconsidering the privacy interests of the deceased in a digital world. Cardozo Arts & Entertainment Law Journal, 32 , 84–126. https://doi.org/10.2139/ssrn.2267388

European Commission. (2021). Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts. Retrieved from https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 . Accessed 15 Oct 2023.

European Parliament News. (2023). EU AI Act: first regulation on artificial intelligence. Retrieved from https://www.europarl.europa.eu/news/en/headlines/society/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence . Accessed 15 Oct 2023.

European Parliament News. (2023). MEPs ready to negotiate first-ever rules for safe and transparent AI. Retrieved from https://www.europarl.europa.eu/news/en/press-room/20230609IPR96212/meps-ready-to-negotiate-first-ever-rules-for-safe-and-transparent-ai . Accessed 15 Oct 2023.

Fagon, J. (2021). The Jessica Simulation: Love and loss in the age of A.I. San Francisco Chronicle . Retrieved from https://www.sfchronicle.com/projects/2021/jessica-simulation-artificial-intelligence/ . Accessed 15 Oct 2023.

Fikfak, V., & Izvorova, L. (2022). Language and persuasion: Human dignity at the european court of human rights. Human Rights Law Review, 22 , 1–24. https://doi.org/10.1093/hrlr/ngac018

Fjeld, J., Achten, N., Hilligoss, H., Nagy, A., Srikumar, M. (2020). Principled artificial intelligence: Mapping consensus in ethical and rights-based approaches to principles for AI. Berkman Klein Center Research Publication No. 2020–1. https://doi.org/10.2139/ssrn.3518482

Fors, K. L., Custers, B., & Keymolen, E. (2019). Reassessing values for emerging big data technologies: Integrating design-based and application-based approaches. Ethics and Information Technology, 21 , 209–226. https://doi.org/10.1007/s10676-019-09503-4

Gach, K. Z., & Brubaker, J. R. (2020). Designing Post-mortem Profile Deletion as a Community Ritual. CHI’20, workshop HCI at End of Life & Beyond, April 25–30, Honolulu, Hi, USA . Retrieved from https://cmci.colorado.edu/idlab/assets/bibliography/pdf/gach-chi2020.pdf . Accessed 15 Oct 2023.

Gibson, M., Carden, C. (2018). Living and dying in a Virtual World. Digital Kinships, Nostalgia and Mourning in Second Life , Palgrave Macmillian. https://doi.org/10.1007/978-3-319-76099-5

Harbinja, E. (2017). Post-mortem privacy 2.0: theory, law, and technology. International Review of Law, Computers & Technology 31, 1: 26–42. https://doi.org/10.1080/13600869.2017.1275116

Harbinja, E. (2019). Posthumous Medical Data Donation: The Case for a Legal Framework. In: J. Krutzinna, L. Floridi (eds.), The Ethics of Medical Data Donation , Philosophical Studies Series , 137, pp. 97–113. https://doi.org/10.1007/978-3-030-04363-6_6

Harbinja, E. (2020). ‘The “New(ish)” Property, Informational Bodies and Postmortality’, in. M Savin-Baden and V Mason-Robbie, (Eds.), Digital Afterlife: Death Matters in a Digital Age , Routledge Taylor & Francis.

Harrison, R. P. (2003). The Dominion of the Dead . The University of Chicago Press.

Hollanek, T. (2020). AI transparency: a matter of reconciling design with critique. AI & Society . https://doi.org/10.1007/s00146-020-01110-y

Huberman, J. (2017). Dearly Departed: Communicating with the Dead in the Digital Age, Social Analysis , 61(3): 91–107. https://www.jstor.org/stable/26593205 . Accessed 15 Oct 2023.

Imagination Lancaster. (2023). What if? Design Fictions for Society: Imagining the future implications of policy initiatives in creative ways. Retrieved 6 November, 2023, from https://imagination.lancaster.ac.uk/project/what-if-design-fictions-for-society/ . Accessed 15 Oct 2023.

Jacobsen, M. H. (Ed.). 2017. Postmortal Society: Towards a Sociology of Immortality , Routledge. https://doi.org/10.4324/9781315601700

Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1 , 389–399. https://doi.org/10.1038/s42256-019-0088-2

Kasket, E. (2019). All the Ghosts in the Machine. Illusion of Immortality in the Digital Age , Robinson, Great Britain.

Kera, D. (2014). Necromedia – Reversed Ontogeny or Posthuman Evolution? In. C. Moreman, D. Lewis (Eds.). Digital Death. Mortality and Beyond in the Online Age, Praeger, California, 181–197.

Konigsberg, R. D. (2011). The Truth About Grief: The Myth of its Five Stages and the New Science of Loss . Simon and Schuster.

Krueger, J., & Osler, L. (2022). Communing with the dead online: Chatbots and continuing bonds. Journal of Consciousness Studies, 29 , 222–252.

Krutzinna, J. and Floridi, L. (2019). Ethical Medical Data Donation: A Pressing Issue. In: J. Krutzinna and L. Floridi (eds.), The Ethics of Medical Data Donation , Philosophical Studies Series , 137. https://www.ncbi.nlm.nih.gov/books/NBK554076 . Accessed 15 Oct 2023.

Lagerkvist, A. (2022). Existential Media . Oxford University Press.

Book   Google Scholar  

Lindemann, N. F. (2022). The Ethics of ‘Deathbots.’ Science and Engineering Ethics, 28 (60), 1–16. https://doi.org/10.1007/s11948-022-00417-x

Lingel, J. (2013). The Digital Remains: Social Media and Practices of Online Grief. The Information Society, 29 (3), 190–195. https://doi.org/10.1080/01972243.2013.777311

Loh, M. (2023). China is using AI to raise the dead, and give people one last chance to say goodbye, Insider . Retrieved from https://www.businessinsider.com/ai-make-money-china-grieving-raise-dead-griefbot-2023-5?r=US&IR=T . Accessed 15 Oct 2023.

Macklin, R. (2003). Dignity is a useless concept. BMJ, 327 , 1419. https://doi.org/10.1136/bmj.327.7429.1419

Mascharka, D., Tran, P., Soklaski, R., & Majumdar, A. (2018). Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018 , 4942–4950. https://doi.org/10.1109/CVPR.2018.00519

Massimi, M., Charise, A. (2009). Dying, death, and mortality: towards thanatosensitivity in HCI. In CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA '09). Association for Computing Machinery, New York, NY, USA, 2459–2468. https://doi.org/10.1145/1520340.1520349

Massimi, M., Odom, W., Banks, R., Kirk, D. ( 2011). Matters of life and death: locating the end of life in lifespan-oriented HCI research. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). Association for Computing Machinery, New York, NY, USA, 987–996. https://doi.org/10.1145/1978942.1979090

Moreman, C., M., Lewis, A. D., (Eds.). 2014. Digital Death. Mortality and Beyond in the Online Age, Praeger, California

Morse, T., & Birnhack, M. (2020). The posthumous privacy paradox: Privacy preferences and behavior regarding digital remains. New Media & Society, 24 (6), 1343–1362. https://doi.org/10.1177/1461444820974955

O’Connor, M., Kasket, E. (2022). What Grief isn’t: Dead grief concepts and their digital-age revival. In T. Machin et al. (Eds.) Social Media and Technology Across the Lifespan, Palgrave Studies in Cyberpsychology., pp. 115–130. https://doi.org/10.1007/978-3-030-99049-7_8

O’Neill, K. (2016). Internet Afterlife. Virtualy Salvation in the 21st Century , Praeger.

Öhman, C., & Floridi, L. (2017). The political economy of death in the age of information: A critical approach to the digital afterlife industry. Minds and Machines: Journal for Artificial Intelligence, Philosophy and Cognitive Science, 27 (4), 639–662. https://doi.org/10.1007/s11023-017-9445-2

Öhman, C., & Floridi, L. (2018). An Ethical Framework for the Digital Afterlife Industry. Nature Human Behavior, 2 , 318–320. https://doi.org/10.1038/s41562-018-0335-2

OpenAI. (2023). Usage policies. Retrieved 7 September, 2023, from https://openai.com/policies/usage-policies . Accessed 15 Oct 2023.

Pitsillides, S. (2019). Digital legacy: Designing with things. Death Studies, 43 (2), 1–9. https://doi.org/10.1080/07481187.2018.1541939

Project December. (2023). Retrieved 7 September, 2023, from https://projectdecember.net . Accessed 15 Oct 2023.

Robitzski, D. (2021). OpenAI Shuts Down GPT-3 Bot Used To Emulate Dead Fiancée. Futurism . Retrieved from https://futurism.com/openai-dead-fiancee . Accessed 15 Oct 2023.

Savin-Badden, M., Burden, D., Taylor, H. (2017). The ethics and impact of digital immortality, Knowledge Cultures 5(2): 178–196. https://eprints.worc.ac.uk/id/eprint/6726 . Accessed 15 Oct 2023.

Sisto, D. (2020). Online Afterlives Immortality, Memory and Grief in Digital Culture, trans. B. McClellan-Broussard, The MIT Press. https://doi.org/10.7551/mitpress/12553.001.0001

Smith, A. ( 2021). Microsoft patent shows plans to revive dead loved ones as chatbots. The Independent . Retrieved from https://www.independent.co.uk/tech/microsoft-chatbot-patent-dead-b1789979.html . Accessed 15 Oct 2023.

Sofka, C. J., Cupit, I., Gilbert, K. R. (Eds.) 2012. Dying, death, and grief in an online universe: For counselors and educators , Springer Publishing Company.

Stokes, P. (2021). Digital Souls: a Philosophy of Online Death . Bloomsbury.

Strubell, E., Ganesh, A., and McCallum Strubell, A. (2019). Energy and Policy Considerations for Deep Learning in NLP. https://doi.org/10.48550/arXiv.1906.02243

Sturdee, M., Coulton, P., Lindley, J.G., Stead, M., Ali, H. and Hudson-Smith, A. (2016). Design Fiction: How to Build a Voight-Kampff Machine. Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16) . Association for Computing Machinery, New York, NY, USA, 375–386. https://doi.org/10.1145/2851581.2892574

Sumiala, J. (2021). Mediated Death . Polity.

Turkle, S. (2011). Alone together. Why We Expect More from Technology and Less from Each Other , NY: Basic Books.

van Wynsberghe, A. (2021). Sustainable AI: AI for sustainability and the sustainability of AI. AI and Ethics, 1 , 213–218. https://doi.org/10.1007/s43681-021-00043-6

Wallace, J., Montague, K., Duncan, K., Carvalho, L. P., Koulidou, N., Mahoney, J., Morrissey, K.,Craig, C., Groot, L. I., Lawson, S., Olivier, P., Trueman, J., Fisher, H. (2020). ReFind: Design, Lived Experience and Ongoingness in Bereavement. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , 1–12, https://doi.org/10.1145/3313831.3376531

Walter, T. (2017). What death means now: Thinking critically about dying and grieving. Policy Press . https://doi.org/10.2307/j.ctt22p7kcp

Weller, A. (2017). Transparency: Motivations and Challenges. Advance online publication. https://doi.org/10.48550/arXiv.1708.01870

Wong, R.Y., Madaio, M. A., and Merrill, N. (2023). Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics. Proc ACM Hum.-Comput. Interact., 7, CSCW1 (April 2023). https://doi.org/10.1145/3579621

Xygkou, A., Siriaraya P., Covaci, A., Prigerson, H. G., Neimeyer, R., Ang C. S., and She, W. J. (2023). The "Conversation" about Loss: Understanding How Chatbot Technology was Used in Supporting People in Grief. Proceedings of CHI’23: Conference on Human Factors in Computing System, Apr 23 –Apr 28, 2023, Hamburg, Germany . ACM, New York, NY, USA. https://doi.org/10.1145/3544548.3581154

Download references

Acknowledgements

We are grateful to the anonymous reviewers whose careful engagement with our work helped us improve this piece. We are also thankful to Stephen Cave who made this work possible.

Our work was funded by the Leverhulme Trust grant no. RC-2015–067 and the Stiftung Mercator GmbH grant no. 200446.

Author information

Authors and affiliations.

Leverhulme Centre for the Future of Intelligence, University of Cambridge, 16 Mill Lane, Cambridge, CB2 1SB, UK

Tomasz Hollanek & Katarzyna Nowaczyk-Basińska

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Tomasz Hollanek .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Hollanek, T., Nowaczyk-Basińska, K. Griefbots, Deadbots, Postmortem Avatars: on Responsible Applications of Generative AI in the Digital Afterlife Industry. Philos. Technol. 37 , 63 (2024). https://doi.org/10.1007/s13347-024-00744-w

Download citation

Received : 17 November 2023

Accepted : 08 April 2024

Published : 09 May 2024

DOI : https://doi.org/10.1007/s13347-024-00744-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Generative AI
• Digital afterlife
• Thanatosensitivity
• Postmortem privacy
• Find a journal
• Publish with us
• Track your research