| Storing Memories as Attractors |
| Demonstration of Hopfield Net |
| Get hopfield from CBN site |
| Example of Pattern Restoration |
| Example of Pattern Restoration |
| Example of Pattern Restoration |
| Example of Pattern Restoration |
| Example of Pattern Restoration |
| Example of Pattern Completion |
| Example of Pattern Completion |
| Example of Pattern Completion |
| Example of Pattern Completion |
| Example of Pattern Completion |
| Example of Association |
| Example of Association |
| Example of Association |
| Example of Association |
| Example of Association |
| Applications of Hopfield Memory |
| Pattern restoration | |
| Pattern completion | |
| Pattern generalization | |
| Pattern association |
| Hopfield Net for Optimization and for Associative Memory |
| For optimization: | ||
| we know the weights (couplings) | ||
| we want to know the minima (solutions) | ||
| For associative memory: | ||
| we know the minima (retrieval states) | ||
| we want to know the weights | ||
| HebbÕs Rule |
| ÒWhen an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth or metabolic change takes place in one or both cells such that AÕs efficiency, as one of the cells firing B, is increased.Ó | |
| ÑDonald Hebb (The Organization of Behavior, 1949, p. 62) |
| Example of Hebbian
Learning: Pattern Imprinted |
| Example of Hebbian
Learning: Partial Pattern Reconstruction |
| Mathematical Model of Hebbian Learning for One Pattern |
| A Single Imprinted Pattern is a Stable State |
| Suppose W = xxT | |
| Then h = Wx = xxTx = nx
since |
|
| Hence, if initial state is s = x, then new state is s¢ = sgn (n x) = x | |
| May be other stable states (e.g., Ðx) |
| Questions |
| How big is the basin of attraction of the imprinted pattern? | |
| How many patterns can be imprinted? | |
| Are there unneeded spurious stable states? | |
| These issues will be addressed in the context of multiple imprinted patterns |
| Imprinting Multiple Patterns |
| Let x1, x2, É, xp be patterns to be imprinted | |
| Define the sum-of-outer-products matrix: |
| Definition of Covariance |
| Consider samples (x1, y1), (x2, y2), É, (xN, yN) |
| Weights & the Covariance Matrix |
| Sample pattern vectors: x1, x2, É, xp | |
| Covariance of ith and jth components: |
| Characteristics of Hopfield Memory |
| Distributed (ÒholographicÓ) | ||
| every pattern is stored in every location (weight) | ||
| Robust | ||
| correct retrieval in spite of noise or error in patterns | ||
| correct operation in spite of considerable weight damage or noise | ||
| Stability of Imprinted Memories |
| Suppose the state is one of the imprinted patterns xm | |
| Then: |
| Interpretation of Inner Products |
| xk × xm = n if they are identical | ||
| highly correlated | ||
| xk × xm = Ðn if they are complementary | ||
| highly correlated (reversed) | ||
| xk × xm = 0 if they are orthogonal | ||
| largely uncorrelated | ||
| xk × xm measures the crosstalk between patterns k and m | ||
| Cosines and Inner products |
| Conditions for Stability |
| Sufficient Conditions for Instability (Case 1) |
| Sufficient Conditions for Instability (Case 2) |
| Sufficient Conditions for Stability |