Incompleteness of moral choice and evolution towards fully autonomous AI

The paper examines whether and how moral competence must transfer to AI systems as tasks requiring human moral authority are automated. It situates the question within the development toward fully autonomous AI platforms—especially autonomous weapons systems (AWS)—and argues that traditional, purely normative ethical frameworks are insufficient for guiding such systems. The aim is to reassess AI ethics by analyzing typical arguments about AWS, scrutinizing assumptions about autonomy, dignity, accountability, free will, and consciousness, and motivating an empirically informed, naturalistic approach to moral decision-making in AI.

The paper surveys: (1) Proponents of artificial moral agents (AMAs) who treat ethics functionally and algorithmically (Allen, Wallach, Anderson). (2) The progression toward autonomous weapons systems (AWS) and classifications by autonomy (human-in/on/out-of-the-loop), with reports from Human Rights Watch and analyses by Williams & Scharre, Boulanin & Verbruggen, Galliott & Lotz, and Sparrow. (3) Three common argumentation strategies against fully autonomous weapons: violations of human dignity and inability of machines to respect life (Docherty; HRW; Heyns; Asaro); accountability gaps when harm occurs (Asaro; Tamburrini); and principled limits of AI due to lack of free will and phenomenal consciousness, leading to unpredictability and error (Nagel; Chalmers; Endsley; Wallach; Deng). (4) A metaethical critique of Kantian deontology and the role of universal moral principles, contrasted with utilitarianism and virtue ethics, arguing these serve as high-level heuristics rather than algorithmic solutions (Kant; Mill; Aristotle; Greene; Wood; Kerstein; Denis). (5) A naturalistic turn: moral intuitions and post hoc reasoning (Haidt); neuroscientific evidence on unconscious bases of moral judgment and perception (Greene et al.; Koch; Ramachandran; Churchland; Gazzaniga; Bear; Wilson), supporting an empirical program for understanding moral decision-making relevant to AI. The review also challenges the subjective/objective dichotomy of values, proposing that both normative and descriptive claims function within broader theoretical practices rather than corresponding to objects in a metaphysical sense.

Conceptual and argumentative analysis grounded in an interdisciplinary narrative review. The author analyzes prevailing argumentation strategies about autonomous AI (especially AWS), critiques Kantian and other principle-based approaches using the "Kant's moral ambulance" metaphor, and advances a naturalistic metaethics informed by findings from psychology and neuroscience (e.g., intuition-driven moral judgment, perceptual and cognitive limitations). No new empirical data are collected; instead, the paper synthesizes existing literature from AI ethics, philosophy, cognitive science, and neuroscience to motivate an empirically oriented framework for AI moral decision-making.

• Automating tasks that require human moral authority implies transferring moral competence to AI; defining that competence requires empirical investigation rather than reliance on purely normative principles.
• Current debates about fully autonomous systems (e.g., AWS) commonly rest on assumptions about dignity, accountability, and ontological gaps (free will/consciousness), but these assumptions do not by themselves yield practical guidance for AI design and governance.
• Kantian, utilitarian, and virtue-ethics principles function as high-level, context-agnostic heuristics ("moral ambulances") that are not algorithmizable and cannot resolve concrete, context-rich moral dilemmas in AI deployment.
• Moral judgments in humans are largely intuition-driven with post hoc rationalization; neuroscientific evidence supports substantial unconscious determinants of moral behavior, suggesting that practical AI ethics should be informed by empirical psychology and neuroscience.
• The subjective/objective dichotomy regarding moral values is unhelpful for AI ethics; values should be treated within broader theoretical practices, analogous to how mathematical or scientific statements gain their utility.
• Therefore, AI ethics should pivot to targeted, empirically grounded measures and cross-disciplinary cooperation (law, cognitive psychology, bioethics, argumentation theory) rather than universal rules, especially for high-stakes autonomy like AWS.

The analysis addresses the central question—what moral competence must be transferred to autonomous AI—by showing that traditional, universalist moral theories cannot be straightforwardly encoded into algorithms for complex, open environments. Recognizing that human moral decision-making is often intuitive and shaped by unconscious processes, the paper argues that AI moral competence should be specified and evaluated through empirical study of decision processes and outcomes in context. This reframes AI ethics from seeking timeless principles to developing practical, testable, and context-sensitive mechanisms, oversight structures, and evaluation methods. The significance lies in redirecting AI ethics toward empirical validation and interdisciplinary practice, which can better handle unpredictability, accountability, and dignity concerns in systems like AWS.

The paper concludes that moral choice for AI is an empirical problem: effective governance and design require targeted, context-sensitive measures rather than flat, universally applicable principles such as Kantian imperatives. It calls for closer collaboration between AI researchers and scholars in ethics, law, cognitive psychology, neuroscience, and related fields to develop practical recommendations and mechanisms for autonomous systems. Future research should: (1) empirically characterize human and AI moral decision processes across contexts; (2) design and test oversight and accountability frameworks for autonomy (especially in AWS); (3) integrate findings from cognitive science and neuroscience to inform AI moral architectures; and (4) move beyond debates on the objectivity/subjectivity of values toward actionable, evaluative practices.

The work is a theoretical and argumentative synthesis without new empirical data; it relies on secondary literature to motivate a naturalistic, empirical program. While AWS serves as a focal example, the paper does not provide concrete implementation frameworks or experimental validations for proposed approaches. The generalizability of its recommendations awaits empirical testing across AI domains and contexts.