Inevitable challenges of autonomy: ethical concerns in personalized algorithmic decision-making

The paper investigates whether and how reliance on personalized algorithmic decision-making (ADM) undermines personal autonomy. It situates ADM within everyday and policy contexts where AI, big data, and machine learning offer efficiency and accuracy but also raise ethical concerns (e.g., manipulation, nudging, and control by platforms). The author contrasts two perspectives: one claiming ADM enhances autonomy by tailoring options and reducing bias; the other warning that ADM can deceive, nudge, and misalign actions with goals. Aligning with the latter, the paper argues that some autonomy harms are inherent and difficult to eliminate. It outlines a plan: review classical decision-making and autonomy, analyze three inherent autonomy challenges in ADM (value-ladenness, self-narrowing via filter bubbles/algorithmic identities, and practical deskilling), assess why common remedies (transparency, awareness, ethics frameworks) are insufficient, and propose a human-in-the-loop strategy to preserve user agency.

The background review covers: (1) Classical decision-making processes (problem identification, information gathering, evaluation, choice, review) and the roles of desires and beliefs under uncertainty; reliance on social networks for value-added, contextualized information; and heuristic versus deliberative systems in human judgment (Kahneman’s fast/slow systems). It notes sources of autonomy threats in traditional settings (value-laden advice, bounded rationality, cognitive biases). (2) Autonomy concepts: independence (freedom from manipulation/control), authenticity (acting from one’s true desires/values), and rational capacity (self-control, critical thinking) (Christman, 2020). User-perceived autonomy in ADM is linked to control/choices, accounting for preferences, transparency/explainability, and data/privacy concerns (Sankaran et al., 2021). Studies suggest perceived autonomy increases with control and explainability and with reduced data demands. (3) Competing views on ADM: proponents argue personalization expands choice sets and mitigates biases, while critics document manipulation, nudging, and misalignment (Yeung, 2017; Burr et al., 2018; Susser et al., 2019). The review also highlights empirical evidence of algorithmic bias (e.g., gender skew in STEM ad delivery), public un/awareness of algorithms, and aversion to machine moral decision-making, setting the stage for the paper’s critical analysis.

Conceptual and normative analysis. The paper synthesizes philosophical accounts of autonomy with scholarship on ADM, algorithmic bias, and human–AI interaction. It constructs arguments about inherent autonomy risks by: (a) analyzing design choices and value trade-offs in algorithm development (targets, metrics, data practices); (b) examining how personalization mechanisms (recommendation loops, filter bubbles) shape identity and exposure to information; (c) assessing cognitive and practical impacts (dependence, deskilling, decision inertia). It uses illustrative real-world examples (platform manipulation, targeted political messaging, biased ad delivery, recommender effects on reading) and draws on empirical studies to support claims. No empirical data collection or experiments are conducted; the contribution is theoretical, offering a mitigation strategy (human-in-the-loop) grounded in user agency.

• Personalized ADM entails intractable autonomy challenges stemming from its mechanisms and human–AI interaction dynamics.
• Value-ladenness of algorithms: platforms can manipulate choice architectures (nudging, deceptive design) to align outcomes with their interests; developers’ target selections and trade-offs encode social norms/preferences; and biased historical data lead to discriminatory outputs (e.g., men targeted more for STEM job ads).
• Users’ self-narrowing: personalization constructs "algorithmic identities"—fragmented, homogenized, datafied profiles that may misrepresent authentic selves; recommender systems create self-reinforcing loops and filter bubbles, reducing exposure to diverse content and constraining genuine choice, undermining substantive independence.
• Practical deskilling and decision inertia: outsourcing reasoning to AIAs fosters dependence, reducing exercise of rational capacities and critical reflection; over time, this degrades practical reason, integral to autonomy and self-constitution.
• Perceived vs. genuine autonomy: users may feel autonomous due to control settings and preference alignment, but subtle manipulation and constrained environments erode cognitive freedom; perceived autonomy can be an illusion.
• Technological remedies (transparency, explainability, governance) and education mitigate risks but cannot eliminate autonomy threats due to ontological differences (AI’s probabilistic simulation of user decision-making) and entrenched social values/biases in data/design.
• Empirical touchpoints: 61% of Norwegians report little/no algorithm awareness (Gran et al., 2021); people are averse to machine-made moral decisions (Bigman & Gray, 2018); evidence of gender bias in ad delivery (Lambrecht & Tucker, 2019); increased perceived autonomy with user control in recommendations (Fink et al., 2024).
• Mitigation strategy: adopt human-in-the-loop interactions placing human judgment centrally, with AI supporting analysis and recommendations; apply adjustable/variable autonomy (human-in/on/out-of-the-loop) based on context to balance efficiency and autonomy risks.

The paper argues that ADM’s autonomy threats—value-ladenness, self-narrowing, and deskilling—are difficult to remove because they flow from: (1) social and cultural biases and designers’ value trade-offs embedded in algorithms; (2) personalization’s inherent feedback loops and user demand for tailored content; and (3) human tendencies toward efficiency and convenience that foster algorithmic dependence. As a result, users often experience perceived autonomy while genuine autonomy is eroded through subtle mental manipulation and reduced cognitive freedom. Recognizing that algorithms can excel at prediction and efficiency does not negate these ethical risks. The author proposes centering user agency through human-in-the-loop designs that keep human judgment, reflection, and ethical reasoning in control, complemented by AI’s data processing strengths. Depending on task criticality and risk, human-on-the-loop or human-out-of-the-loop modes, and adjustable autonomy, can be selected to pragmatically balance benefits and autonomy. Enhancing algorithm awareness can help users maintain healthy skepticism and align their practices with a tool-centric view of AI, mitigating autonomy erosion.

The article identifies three inherent autonomy challenges in personalized ADM—value-ladenness, narrowing of self through algorithmic identities and filter bubbles, and practical deskilling—and argues these stem from intrinsic features of personalization and human–AI interaction. Technical and policy measures can reduce but not eliminate these risks. To safeguard autonomy, the paper recommends a human-in-the-loop approach that positions AI as a complement to human judgment, coupled with adjustable autonomy tailored to context and efforts to raise users’ algorithm awareness. This framework aims to preserve agency and authentic decision-making while leveraging AI’s analytical strengths.

• Conceptual/normative analysis without original empirical testing; claims rely on synthesis of literature and illustrative cases.
• Does not specify or evaluate concrete implementations of human-in-the-loop or Socratic AI designs; practical pathways are noted as beyond scope.
• Generality across diverse platforms, cultures, and application domains is assumed but not empirically validated.
• The extent of deskilling and autonomy erosion may vary by context and user characteristics; the paper does not quantify these effects.