Aristotle observed that human memory has the ability to connect items (e.g. objects, feelings and ideas) that are similar, contradictory, that occur in close proximity, or in succession [Kohonen 1987]. The patterns that we associate may be of the same, or different types. For example, a photo of the sea may bring associated thoughts of happiness, or smelling a specific fragrance may be associated with a certain feeling, memory or visual image. Also, the ability to reproduce the pitch corresponding to a note, irrespective of the form of the note, is an example of the pattern association behavior of the human brain.

Artificial neural networks have been developed to model the pattern association ability of the human brain. These networks are referred to as associative memory NNs. Associative memory NNs are usually two-layer NNs, where the objective is to adjust the weights such that the network can store a set of pattern associations - without any external help from a teacher. The development of these associative memory NNs is mainly inspired from studies of the visual and auditory cortex of mammalian organisms, such as the bat. These artificial NNs are based on the fact that parts of the brain are organized such that different sensory inputs are represented by topologically ordered computational maps. The networks form a topographic map of the input patterns, where the coordinates of the neurons correspond to intrinsic features of the input patterns. An additional feature modeled with associative memory NNs is to preserve old information as new information becomes available. In contrast, supervised learning NNs have to retrain on all the information when new data becomes available; if not, supervised networks tend to focus on the new information, forgetting what the network has learned already. Unsupervised learning NNs are functions which map an input pattern to an associated target pattern, i.e. FNN : M7 -> R* (4.1) as illustrated in Figure 4.1. The single weight matrix determines the mapping from the input vector z to the output vector o.

The Hebbian learning rule, named after the neuropsychologist Hebb, is the oldest and simplest learning rule. With Hebbian learning, weight values are adjusted based on the correlation of neuron activation values. The motivation of this approach is from Hebb's hypothesis that the ability of a neuron to fire is based on that neuron's ability to cause other neurons connected to it to fire. In such cases the weight between the two correlated neurons is strengthened (or increased). Using the notation from Figure 4.1, the change in weight at time step t is given as ) = r]ok,pzi,p Weights are then updated using uki(t) = uki(t - 1) + &uki(t) where 77 is the learning rate. From equation (4.2), the adjustment of weight values is larger for those input-output pairs for which the input value has a greater effect on the output values. (4.3) (4.2)

Figure 4.1: Unsupervised neural network A summary of the Hebbian learning rule is given below: 1. Initialize all weights such that uki = 0, Vi = 1, , / and VA; = 1, , K. 2. For each input pattern zp compute the corresponding output vector op. 3. Adjust the weights using equation (4.3). 4. Stop when the changes in weights are sufficiently small, or the maximum number of epochs has been reached; otherwise go to step 2. A problem with Hebbian learning is that repeated presentation of input patterns leads to an exponential growth in weight values, driving the weights into saturation. To prevent saturation, a limit is posed on the increase in weight values. One type of limit is to introduce a nonlinear forgetting factor.

' - 1)

(4-4)

where a is a positive constant, or equivalently, uki(t - 1)]

(4.5)