Unsupervised Learning in Neural Networks

The Hebbian Principle

Thus far we have described the dynamics of the Hopfield network but have not actually dealt with the way weights are established. As early as 1949 Donald Hebb proposed that one way biological neurones learn associations in the real world is that correlation of activity between two neurones is reinforced by increasing synaptic strength between them. In this way a neurone firing with the sight of an apple might become associated with another neurone firing at the smell of an apple such that one leads to another. Put another way, those neurones that fire together wire together, while those that fire out of sync lose their link.

In his original paper (1982) Hopfield did not give a method of training the nets but instead gave a "prescription" for making a weight set, given a set of patterns to be stored. The prescription is derived from Hebb's law given above. The idea of this prescription was to use the weights to capture the correlation between node outputs when the net is in one of the required stable states. For example, two nodes that, on average over the required training set, tend to be on the same value will tend to be connected by a positive weight. Conversely, two nodes that tend to take on opposite values will tend to be connected by a negative weight.

This is described mathematically in the following way: first, rather than using 0 or 1 for output it is convention to use the -1 or 1.

vi = {-1,1}

Now let vip and vjp be units in the pth pattern to be stored. Consider what happens when the weight between the nodes I and j is:

If, on average, the two units take on the same value the weight will be positive whereas if the units take opposite values we get negative weights predominating.

In addition to the above prescription, other learning rules can also be applied to the Hopfield network including the delta rule described under supervised learning rules.

A final note about the storage capacity of Hopfield's net. As the number of patterns increases the chances of accurate storage must decrease. Hopfields empirical work in 1982 showed that about half of the memories were stored accurately in a net of N neurones if the number of patterns was 0.15N. McCliece performed a more rigorous analysis in 1987and showed that if almost all memories needed to be stored accurately then the maximum number of patterns is N/2logN. So, for a network consisting of about 100 nodes can remember about 10 patterns. Thus, the number of patterns that can be stored is limited in Hopfield networks. Another problem with Hopfield networks is that patterns that share many features in common with each other tend to become unstable.

Previous | Next | Page 1 2 3 4

Index | Unsupervised Learning | The Hebbian Principle

References:

Hebb, D. The Organization of behaviour. John Wiley, 1949.

J.J. Hopfield. Neural networks and physical systems with emergent collective computational properties. Proceedings of the National Academy of Sciences of the USA, 79:2554 - 2588, 1982.

McEliece, R., Posner, E., Rodemich, E., and Venkatesh, S. The capacity of the hopfield associative memory. IEEE Transactions on Information Theory, IT-33:461 - 482, 1987.

© 2008 Marcus bros

Message Board released We have just opened a new message board that will provide a centre for... 29 Jul 2005 by marcus comments (0) Website released We have released our new website to provide information about our... 29 Jul 2005 by marcus comments (0)