Self-mediated Exploration in Artificial Intelligence Inspired by Cognitive Psychology

The study addresses how epistemic (e.g., surprise) and achievement (e.g., pride) emotions can mediate exploratory behavior in artificial agents, inspired by extensive findings in cognitive psychology linking these emotions to exploration and learning. The authors argue that AI systems lack intrinsic exploratory drives comparable to biological organisms and propose explicitly modeling internal emotional states derived from task performance metrics (accuracy and confidence). They hypothesize that replicating cognitive conditions that elicit surprise (e.g., high-confidence errors) and pride (success/accuracy) in humans will causally influence exploration in AI. They design an AI experiment mirroring human trials to test whether agents can learn a relationship between these emotions and exploration, aiming to validate psychological findings and enhance AI autonomy and adaptability.

The paper situates its contribution within cognitive psychology literature showing that internal cognitive conditions (incongruity, expectancy, control-value appraisals) generate epistemic and achievement emotions, which in turn influence curiosity, confusion, motivation, and exploration. Prior psychology studies (e.g., Vogl et al.) found that surprise, curiosity, and confusion robustly promote knowledge exploration. In contrast, many AI approaches induce exploration via intrinsic motivations (model prediction error differences, divergence in transition probabilities, intrinsic rewards) without explicitly modeling the originating emotional conditions. The authors emphasize aligning AI emotion functions with psychological theory: pride positively associated with achievement/success (accuracy), and surprise elicited by unexpected outcomes, particularly high-confidence errors or low-confidence successes (requiring both accuracy and confidence). They also highlight the free-energy principle as a normative account for minimizing uncertainty, consistent with surprise reduction.

Task and data: The MNIST dataset is used. The training set (60,000 images) is split: half used to pre-train the task-oriented classifier; the other half (30,000 images) is used in the main experiments after adulteration so that 50% of labels are randomly incorrect. The test set (10,000 images) assesses pre-training performance. Task-oriented module: A simple VGG-like CNN (two conv layers with 32 and 64 filters, stride 3, each followed by max pooling; 0.5 dropout; two-layer perceptron classifier) is trained for 50 epochs with Adam (default params). It achieves 99.2% test accuracy with 0.03 loss, implying high confidence. Emotion functions: Achievement emotion (pride) P depends on accuracy a ∈ [0,1] via a positively sloped, bounded, noisy function, e.g., P(a) = Clip((100·C1)^(-(a−1)^2) + N(0,0.03)), with C1>1. Epistemic emotion (surprise) S depends on accuracy a and confidence c ∈ [0.8,1] via a saddle-like surface capturing high surprise for high-confidence errors and low-confidence successes, e.g., S(c,a) = Clip(T R (a^2 − c^2) + 0.5 + N(0,0.03)), with rotation R (±C2, C2∈[−20°,20°]) and translation T. Noise introduces inter-individual variability and clipping bounds outputs to [0,1]. RL architecture: An actor-critic setup using deterministic policy gradients (DPG/DDPG) maps emotional state s to an exploratory rate a. Actor μθ(s) outputs exploration rate; critic Qω(s,a) evaluates the action. Replay buffer stores tuples (s,a,r,s′); mini-batch size 64. Target networks μ′ and Q′ are used with soft updates τ=0.005. Discount γ=0.99. Actor learning rate 0.001; critic learning rate 0.002. Optimization via Adam. The critic receives both the emotional score and the actor-chosen exploration rate (separate branches merged into an MLP); the actor is an MLP with one hidden layer. Learning cycle (experiment): - Phase 1: Pre-train the task-oriented classifier to high accuracy. - Phase 2: Main emotion–exploration learning with adulterated data. At each step, a single image is sampled and classified; due to label adulteration, agents experience both correct outcomes and high-confidence errors. Emotional score (surprise or pride) is computed from accuracy (and confidence, for surprise). The actor maps emotion to an exploration rate, determining the size (up to 64) of a same-type batch for subsequent analysis. Emotional variation between single-instance and batch steps is tracked. - Rewards: (i) Performance-based reward proportional to the difference between batch accuracy and single-instance accuracy (encouraging exploration only if it improves task performance). (ii) Sparse emotion-based reward encouraging surprise minimization or pride maximization, aligned with the free-energy principle. Agents and training regime: Separate experiments are conducted for surprise and pride with 250 distinct agents each, introducing random noise in emotion functions and action selection to model individual differences. Each agent runs 100 episodes of 20 steps; systems reset every 20 steps (episodes), following a within-person design comparable to Vogl et al. Confidence scores are sampled high (0.8–1.0). Computation uses Keras/TensorFlow with an NVIDIA RTX 2080 GPU.

- Convergence: Agents learned stable policies. Cumulative reward increased and plateaued over episodes for both emotions, indicating successful learning of emotion-to-exploration mappings. - Emotion dynamics: Surprise bursts became 38.52% less frequent by the final episode, consistent with surprise minimization/free-energy reduction. Pride showed a mild upward tendency, with pride decreases occurring 5.90% fewer times. - Surprise → exploration: Across 250 agents, exploration increased with surprise by an average of 15.4%. 217 agents learned a positive monotonic relationship; 33 learned negative. The learned relationships were monotonic and often substantial. - Pride → exploration: Overall, exploration decreased slightly (−2.8%) as pride increased. Although 222 agents showed slight positive relationships, their effects were weak; 22 agents showed strong negative relationships (−25% to −75% toward null), and 6 showed moderate negative effects, yielding a net average decrease. - Correlation strength over time: Episode-wise Spearman correlations between emotion and exploration started near zero and evolved to a moderate positive value for surprise (smoothed ρ ≈ 0.461 by the end; non-windowed mean across agents ≈ 0.311 in the final episode) and remained weakly negative for pride (near zero but below, indicating a weak dampening effect). - Human-aligned replication: The AI results mirror psychology findings (Vogl et al.), with robust positive links for surprise and weak/ambiguous, often dampening, effects for pride on exploration.

The findings demonstrate that explicitly modeled epistemic and achievement emotions can causally mediate exploration in artificial agents, aligning AI behavior with human psychological evidence. Surprise robustly drives exploration and is progressively minimized, consistent with the free-energy principle. Pride exerts, at most, a weak and often negative influence on exploration. Architecturally, an actor-critic mechanism inspired by basal ganglia circuits can successfully translate internal emotional states into adaptive exploratory control, improving task performance. The approach bridges cognitive psychology and AI by recreating experimental conditions under which human emotion–exploration links have been observed, offering a platform to test and corroborate hypotheses about human cognition and learning. Compared to AI intrinsic motivation literature, this method explicitly re-creates emotion-inducing conditions and embeds psychologically grounded emotion functions, advancing the interpretability and biological plausibility of exploration drivers.

This work provides a proof-of-concept that artificial agents can learn self-mediated exploration driven by psychologically grounded emotions. By embedding surprise and pride functions derived from task performance and confidence, and learning emotion-to-exploration mappings via actor-critic RL, agents replicated human-like patterns: strong positive links between surprise and exploration and weak, often dampening, links for pride. Contributions include: (i) explicit implementation of epistemic and achievement emotions in AI, (ii) an AI experimental framework mirroring human cognitive psychology studies, and (iii) evidence supporting interdisciplinary use of AI to study cognition. Future directions: model overlapping/multiple emotions jointly (e.g., combining epistemic and achievement states), extend to other behavioral traits (exploitation/engagement), generalize beyond MNIST to diverse tasks and modalities, and use such agents to probe neural hypotheses about emotion, attention, and executive function.

- Emotions were studied individually; real behavior involves overlapping emotional states with joint effects. - The task and setup differ from human experiments (image classification with label adulteration vs. trivia/veracity tasks), limiting direct comparability. - Confidence was simulated rather than inferred from a calibrated model, and label adulteration is an artificial manipulation. - The actor-critic architecture is only a loose analogy to basal ganglia; neurophysiological fidelity is limited. - Reported correlation metrics for pride remain near zero and incompletely specified in one section, indicating weaker or variable effects. - Results are shown on MNIST with a simple CNN; generalization to more complex tasks and environments remains to be tested.