This paper explores the possibility that the computational abilities of large collections of neurons, such as memory and categorization, are emergent collective properties arising from the interactions of many simple neurons. The author argues that evolutionary processes, unlike human engineering, do not follow a pre-defined plan, making the spontaneous emergence of computation a relevant question. Existing models of neural function often focus on small circuits of neurons that produce elementary behaviors, which can be scaled up in electronics, but this approach is less relevant to understanding biological systems. The paper proposes a new model to investigate this question, aiming to identify collective properties robust against changes in model details and readily implementable in integrated circuits. The model's findings suggest the design of a distributed content-addressable memory using asynchronous parallel processing.

The paper reviews previous work on neural networks and associative memory, including models by McCulloch and Pitts, Perceptrons, and linear associative nets. It notes limitations of previous models: Perceptrons focused on unidirectional connections, and their analysis with strong backward coupling proved difficult. Linear associative nets, while capable of storing multiple associations, produced meaningless mixed outputs when presented with ambiguous inputs. Existing models often involved synchronous processing, a contrast to the asynchronous nature of biological neural systems. The author differentiates their model from previous efforts such as those by Little, Shaw, and Roney, which used 'on/off' neurons and synchronous processing, focusing on the timing of action potentials, while this model emphasizes the nonlinear relationship between neuronal input and output in the context of asynchronous processing.

The model proposes a system of interconnected processing devices called neurons, each existing in two states: 'firing' (V=1) and 'not firing' (V=0). The connection strength between neurons i and j is represented by T_ij. The state of each neuron changes asynchronously and randomly based on a threshold U_i and the weighted sum of inputs from other neurons: V_i → 1 if ΣT_ijV_j > U_i, and V_i → 0 otherwise. This differs from Perceptrons in its use of strong back-coupling and asynchronous processing. The algorithm for storing memories involves a Hebbian-like rule: T_ij = Σ(2V^s_i − 1)(2V^s_j − 1), where V^s represents the state vector of memory s. The analysis employs concepts from statistical mechanics and utilizes Monte Carlo simulations on systems of N=30 and N=100 neurons. The simulations assess the stability of stored memories, the effect of noise, the capacity of the network, and the impact of factors such as asymmetry in the connection matrix, clipping of synaptic weights, and the addition of new memories. Statistical measures like Hamming distance and entropy are used to characterize the system's behavior. The system is also analyzed in terms of an energy function, which provides a measure of the system's stability and helps in understanding the nature of the stable states.

The model demonstrates that a network of simple interconnected neurons with an asynchronous update rule can exhibit complex emergent computational properties. The simulations reveal that the network functions as a content-addressable memory, capable of retrieving complete memories from partial cues. The model exhibits error correction and can categorize and generalize from inputs. For N = 30 neurons, the system usually settled into a stable state or a simple cycle within a short time. The stability of stored memories depends on the number of memories stored relative to the number of neurons (n/N). The network showed a capacity to store approximately 0.15N memories with acceptable error rates. Simulations showed that the system could retain memories even when the connection matrix T_ij was not symmetric (T_ij ≠ T_ji), indicating robustness to variations in network structure. The addition of a uniform threshold allows for the recognition of unfamiliar inputs. The model also demonstrates limited capacity for the temporal ordering of memories. Experiments with 'clipped' T_ij (replacing T_ij with its sign) showed similar performance, suggesting robustness to nonlinearities in synaptic weights. The model also displays some degree of soft-failure tolerance, with performance degrading gracefully as individual units fail. Memories that are too close together in Hamming distance tend to merge. The simulations demonstrate how the initial processing rate can be used to distinguish familiar from unfamiliar states even when the network is heavily overloaded.

The model's success in exhibiting complex computational abilities from simple components suggests a mechanism for the emergence of intelligence in biological systems. The robustness of the collective properties to details of the model indicates that the observed phenomena are not mere artifacts of specific assumptions, but rather reflect fundamental principles of interacting dynamical systems. The asynchronous parallel processing nature of the model is computationally efficient and is more biologically plausible than synchronous models. The soft-failure property suggests that this model could lead to more robust hardware implementations of memory systems.

This paper presents a novel model of neural networks that demonstrates the emergence of complex computational properties from simple, interacting components. The model is robust to variations in network parameters and exhibits content-addressable memory, error correction, generalization, categorization, and limited time sequence retention. This research opens new avenues for designing robust, fault-tolerant computational systems and provides insights into the potential mechanisms underlying information processing in biological brains. Future research could explore the model's capacity for handling longer time sequences, investigate the impact of more realistic neuronal models, and develop more sophisticated learning rules.

The model's scale (N=30 and N=100) is relatively small compared to the complexity of real brains. While the robustness suggests that the results will generalize to larger networks, further investigation is warranted. The model simplifies biological neurons; it neglects many features of real neurons, such as graded responses, diverse neuronal types, and complex synaptic dynamics. The model's learning rule is a simplification of biological learning processes. The model's capacity for temporal memory is limited, suggesting potential future research directions.