globals [ max-X ; X coordinates are +- max-X max-Y ; Y coordinates are +- max-Y test-datum ; the current datum under test weight ; the current weight vector new-weight ; temporary new weight vector weight-change? ; set if a weight has changed learned-angle ; the angle of the currrent weight vector ] breed [ data datum ] ; type of the data points breed [ origins origin ] ; type of the origin point breed [ weight-vectors weight-vector ] ; type of the weight vector directed-link-breed [ normal-vectors normal-vector ] ; normal vector to separator undirected-link-breed [ separatrices separatrix ] ; separator line segments breed [ septx-ends septx-end ] ; end points of separator directed-link-breed [ data-vectors data-vector ] ; visible data vectors data-own [ positive? ] ; set if it is a positive example septx-ends-own [ theta ] ; angle separator ends (+- 90) to setup ca set max-X max-pxcor - 1 set max-Y max-pycor - 1 create-origins 1 [ initialize-origin ] create-data num_data_points [ initialize-data ] create-weight-vectors 1 [ initialize-weight-vector ] create-septx-ends 1 [ initialize-separatrix 90 ] create-septx-ends 1 [ initialize-separatrix -90 ] set weight-change? false set learned-angle [ atan xcor ycor ] of weight end to initialize-origin set size 2 set shape "circle" set color green setxy 0 0 end to initialize-data set size 2 set shape "circle" set color red set xcor (random (2 * max-X) - max-X) set ycor (random (2 * max-Y) - max-Y) ifelse non-linear-separable <= random 100 ; generate lin-sep data [ ifelse (xcor * sin correct_separator + ycor * cos correct_separator) > 0 [ set positive? true ] [ set positive? false ] ] [ ifelse 0 = random 1 ; generate non-lin-sep data [ set positive? true ] [ set positive? false ] ] set color ifelse-value positive? [ red ] [ blue ] end ;; Generate initial random weight vector ;; Weight-vector procedure to initialize-weight-vector hide-turtle let init-theta random 360 let magnitude 0.9 * max-X setxy (magnitude * sin init-theta) (magnitude * cos init-theta) create-normal-vector-from origin 0 [ set thickness 0.5 set color green ] set weight self end ;; Generate separator line segments relative to weight vector ;; Separatrix endpoint (septx-end) procedure to initialize-separatrix [init-theta] ; theta = +- 90 hide-turtle set theta init-theta move-separatrix create-separatrix-with origin 0 [ set thickness 0.5 set color green ] end ;; Reposition separatrix endpoint relative to weight vector ;; Separatrix endpoint (septx-end) procedure to move-separatrix setxy 0 0 set heading theta + towards weight fd max-X end ;; Convert negative examples to positive examples to reflect-negatives let w_X [ xcor ] of weight let w_Y [ ycor ] of weight ask data [ if not positive? [ setxy (- xcor) (- ycor) ] ] wait 0.5 ; so that color change is visible ask data [ set color red ] end ;; Cycle through learning procedure steps to go test-one wait delay calculate-new-weight wait delay update-weight wait delay end ;; Select random datum for testing to test-one set test-datum one-of data ask test-datum [ set color yellow create-data-vector-from origin 0 [ set color yellow ] ] set weight-change? false end ;; Calculate new weight vector (if necessary) for test datum to calculate-new-weight let old_X [ xcor ] of weight let old_Y [ ycor ] of weight let datum_X [ xcor ] of test-datum let datum_Y [ ycor ] of test-datum if (old_X * datum_X + old_Y * datum_Y) <= 0 [ create-weight-vectors 1 [ ; create temporary new weight set new-weight self setxy (old_X + eta * datum_X) (old_Y + eta * datum_Y) create-data-vector-from weight [ ; the update vector set color yellow ] create-data-vector-from origin 0 [ ; the new weight vector set color blue ] ] set weight-change? true ] end ;; Update weight vector and update display to update-weight ask test-datum [ set color red ] ask data-vectors [ die ] if weight-change? [ ask weight [ move-to new-weight ] ask new-weight [ die ] ; destroy temporary new weight ask septx-ends [ move-separatrix ] set weight-change? false set learned-angle [ atan xcor ycor ] of weight ] end @#$#@#$#@ GRAPHICS-WINDOW 190 10 588 429 48 48 4.0 1 10 1 1 1 0 0 0 1 -48 48 -48 48 0 0 1 ticks SLIDER 10 10 180 43 num_data_points num_data_points 0 100 10 1 1 NIL HORIZONTAL SLIDER 10 45 180 78 correct_separator correct_separator 0 359 50 1 1 deg HORIZONTAL BUTTON 10 125 180 158 setup setup NIL 1 T OBSERVER NIL NIL NIL NIL BUTTON 10 395 180 428 go go T 1 T OBSERVER NIL NIL NIL NIL BUTTON 10 245 180 278 test one datum test-one NIL 1 T OBSERVER NIL NIL NIL NIL BUTTON 10 280 180 313 calculate new weight calculate-new-weight NIL 1 T OBSERVER NIL NIL NIL NIL BUTTON 10 315 180 348 update weight update-weight NIL 1 T OBSERVER NIL NIL NIL NIL SLIDER 10 205 180 238 eta eta 0 1 0.1 0.01 1 NIL HORIZONTAL SLIDER 10 360 180 393 delay delay 0 1 0.2 0.01 1 sec. HORIZONTAL BUTTON 10 160 180 193 reflect negatives reflect-negatives NIL 1 T OBSERVER NIL NIL NIL NIL SLIDER 10 80 180 113 non-linear-separable non-linear-separable 0 10 0 1 1 % HORIZONTAL MONITOR 10 435 180 480 learned angle learned-angle 1 1 11 @#$#@#$#@ WHAT IS IT? ----------- This model demonstrates the geometry of the Perceptron Learning Algorithm. It generates a linearly separable, or almost linearly separable, set of data, and shows how a weight vector can be adjusted so that a single perceptron is able to separate the positive and negative data points. Two classes of data are linearly separable if they can be separated by a straight line, flat plane, or a flat hyperplane (for dimensions greater than 3). A perceptron is a simple artificial neuron with n inputs and one binary output. Each input k has a weight (synaptic strength) w_k, and so we can treat the n weights as a vector, w = (w_1, ..., w_n). Each neuron also has a threshold t, and if the weighted inputs are greater than the threshold, the neuron produces a 1 output, otherwise it produces a 0 output. Therefore, if the input is a vector x = (x_1, ..., x_n), then the output will be 1 if w_1 x_1 + ... + w_n x_n > t, and 0 otherwise. In vector notation, we can say the output is 1 if the dot product w.x > t. Suppose we are given a dataset comprising a number p of data vectors x1, x2, ..., xp, and that some of these are positive examples of some class of patterns (i.e., they fit the pattern), and others are negative examples (they don't fit). The perceptron learning problem is to find a single threshold and set of weights so that the perceptron correctly classifies all these data (and, hopefully, other data of a similar kind). The perceptron learning algorithm solves this problem. There are several things we can do to simplify perceptron learning. First, we can eliminate the threshold by treating it as an extra, "zeroth" weight. To do this, we add a corresponding zeroth element to the data vectors, which is always -1. In effect the zeroth input is "clamped" at a -1 value. To see why this works, define an extended weight vector W where W_0 = t, W_1 = w_1, ..., W_n = w_n. Also, let X be the "extended" vector corresponding to x: X = (-1, x_1, x_2, ..., x_n). So now we are working with n+1 dimensional vectors. Note that: W.X = t (-1) + w_1 x_1 + ... + w_n x_n = w.x - t Thus w.x > t if and only if W.X > 0. Therefore, since the threhold has been turned into a weight, we only have to worry about adjusting weights. The second simplification is to find a way to treat the positive and negative examples the same way. If X is a positive example, we want W.X > 0, but if X is a negative example, we want W.X < 0. However, since W.(-X) = -W.X, we know that W.X < 0 if and only if W.(-X) > 0. Therefore, we can replace the negative examples by their complements and treat them like positive examples. Specifically, we will define a new set of test data Z1, ..., Zp corresponding to the original data X1, ..., Xp. If Xi is a positive sample, then Zi = Xi, but if Xi is a negative sample, then Zi = -Xi. Therefore, we have a simplified perceptron training problem, which is to find a weight wector W such that W.Z > 0 for every (modified) sample data vector Z. For ease in visualization, this model uses two-dimensional data and weight vectors. It generates random test data, linearly separable into positive and negative examples. The model then converts all the data to positive example, as explained above, and demonstrates the adjustment of the weight vector so that all the test data are on the same (positive) side of the separating line. HOW IT WORKS ------------ The learning procedure is as follows. The algorithm selects a random data point. If it is already classified correctly (i.e., W.Z > 0), the algorithm does nothing. If it is classified incorrectly (W.Z <= 0), then it alters the weight vector (to W') so that the classification is closer to being correct (W'.Z > W.Z). It does this by vector addition of a fraction (eta) of the data vector to the weight vector in order to produce the new weight vector: W' = W + eta * Z. The model displays this vector addition, which can be seen most clearly by stepping through the update process (clicking repeatedly in order: Test One Datum, Calculate New Weight, and Update Weight). Eta determines the learning rate, but if it is too high, it may lead to instability, since the weight vector may adapt too much to the most recent datum, and "forget" previous ones. HOW TO USE IT ------------- NUM DATA POINTS determines the number of random test data points to be generated. They have an equal porbability of being positive (shown in red) or negative (shown in blue). The two classes are all or mostly linearly separable. CORRECT SEPARATOR set the angle of the normal vector to the line through the origin that will separate the positive from the negative samples (except for the exceptions: see next). NON-LINEAR-SEPARABLE determines the percentage of samples that are not linearly separable. They are located randomly and are randomly positive or negative. Set this slider to 0 in order to have the positive and negative samples separated by Correct Separator. SETUP generates the requested number of sample data points, classified according to the specified separator and percentage of exception. Positive and negative samples are indicated by red and blue, respectively. The initial, randomly chosen weight vector is shown in green, along with the separating line perpendicular to it. REFLECT NEGATIVES converts all the samples to positive samples by replacing the negative samples by their negations. After Setup, you should click Reflect Negatives so that all the data are on one side of the separator. ETA is the learning rate. It can be changed while the model is running. For example decreasing Eta may allow a non-converging learning process to converge. To walk through learning process step by step, use the following three buttons: TEST ONE DATUM picks a random data point for learning; it is indicated by yellow color. CALCULATE NEW WEIGHT calculates the new weight vector (blue), which is shown as a vector sum of the old weight vector (green) and Eta times the test datum (yellow). UPDATE WEIGHT replaces the old weight vector by the new one, and rotates the separating line accordingly. The following controls allow the learning algorithm to be run continuously: DELAY is the amount of time between each of the above three steps in continuous update (GO) mode. You can change the Delay while the model is running in order to speed it up or slow it down. GO continuously cycles through the three steps, Test One Datum, Calculate New Weight, and Update Weight. LEARNED ANGLE displays the angle of the learned weight vector, which may be compared to the angle set by Correct Separator. THINGS TO NOTICE ---------------- Notice that if the learning rate Eta is too high, the learning process may not converge. Try decreasing Eta to see if it converges. Observe the differences depending on whether the initial random weight vector tends to point in about the same direction as the positive test, or tends to point in the opposite direction. Observe the effects of non-linearly separable data on learning convergence. Notice how close the angle of the learned weight vector comes to approximate the angle of the separator used to generate the test data. Can you think of ways of altering the learning algorithm to make the approximation closer? THINGS TO TRY ------------- Try various learning rates (Eta). The Perceptron Learning Theorem proves that for linearly separable data the Perceptron Learning Algorithm will always converge *if the learning rate is sufficiently slow*. On the other hand, we would like learning to go as fast as possible. Run some experiments and determine a learning rate that always, or almost always, converges. Does it depend on the number of data points? Try introducing a small percentage (e.g., 5%) of non-linearly separable data, and observe the behavior of the model with several randomly generated datasets. EXTENDING THE MODEL ------------------- Notice that the weight vector often gets shorter as it is updated, which could lead to numerical problems with a large number of updates. Since only the direction of the weight vector matters, how could you modify the model keep it from getting shorter? Since all that really matters is the directions of the vectors, and not their lengths, consider modifying the algorithm to work in terms of normalized vectors. How does this new algorithm compare with the original one. The original perceptron learning algorithm, embodied in this model, makes no weight adjustment for data that are classified correctly ("if it ain't broke, don't fix it"). Can you think of some useful adjustment that could be done for correctly classified data? Try out your idea. NETLOGO FEATURES ---------------- This model uses the links feature to draw the vectors and the left and right segments of the separating line. The endpoints of these lines are represented by turtles (sometimes hidden). RELATED MODELS -------------- There are two related models in the standard Models Library (Computer Science, unverfied): Perceptron, which gives a neural network view of perceptron learning, and Artificial Neural Net, which demonstrates the back-propagation learning (essentially multilayer perceptron learning). CREDITS AND REFERENCES ---------------------- The perceptron, which was one of the first artificial neural net models, was developed by Frank Rosenblatt in the late 1950s. It is described in most textbooks on artificial neural networks, but this particular model follows Ballard (1997, ¤8.2). Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65, 386-408. Rosenblatt, F. (1962). Principles of neurodynamics. Spartan Books. Ballard, D.H. (1997). An introduction to natural computation. MIT Press. To refer to this model in academic publications, please use: MacLennan, B.J. (2008). NetLogo Perceptron Geometry model. http://www.cs.utk.edu/~mclennan. Dept. of Electrical Engineering & Computer Science, Univ. of Tennessee, Knoxville. In other publications, please use: Copyright 2008 Bruce MacLennan. All rights reserved. See http://www.cs.utk.edu/~mclennan/420/NetLogo/Perceptron-Geometry.html for terms of use. @#$#@#$#@ default true 0 Polygon -7500403 true true 150 5 40 250 150 205 260 250 airplane true 0 Polygon -7500403 true true 150 0 135 15 120 60 120 105 15 165 15 195 120 180 135 240 105 270 120 285 150 270 180 285 210 270 165 240 180 180 285 195 285 165 180 105 180 60 165 15 arrow true 0 Polygon -7500403 true true 150 0 0 150 105 150 105 293 195 293 195 150 300 150 box false 0 Polygon -7500403 true true 150 285 285 225 285 75 150 135 Polygon -7500403 true true 150 135 15 75 150 15 285 75 Polygon -7500403 true true 15 75 15 225 150 285 150 135 Line -16777216 false 150 285 150 135 Line -16777216 false 150 135 15 75 Line -16777216 false 150 135 285 75 bug true 0 Circle -7500403 true true 96 182 108 Circle -7500403 true true 110 127 80 Circle -7500403 true true 110 75 80 Line -7500403 true 150 100 80 30 Line -7500403 true 150 100 220 30 butterfly true 0 Polygon -7500403 true true 150 165 209 199 225 225 225 255 195 270 165 255 150 240 Polygon -7500403 true true 150 165 89 198 75 225 75 255 105 270 135 255 150 240 Polygon -7500403 true true 139 148 100 105 55 90 25 90 10 105 10 135 25 180 40 195 85 194 139 163 Polygon -7500403 true true 162 150 200 105 245 90 275 90 290 105 290 135 275 180 260 195 215 195 162 165 Polygon -16777216 true false 150 255 135 225 120 150 135 120 150 105 165 120 180 150 165 225 Circle -16777216 true false 135 90 30 Line -16777216 false 150 105 195 60 Line -16777216 false 150 105 105 60 car false 0 Polygon -7500403 true true 300 180 279 164 261 144 240 135 226 132 213 106 203 84 185 63 159 50 135 50 75 60 0 150 0 165 0 225 300 225 300 180 Circle -16777216 true false 180 180 90 Circle -16777216 true false 30 180 90 Polygon -16777216 true false 162 80 132 78 134 135 209 135 194 105 189 96 180 89 Circle -7500403 true true 47 195 58 Circle -7500403 true true 195 195 58 circle false 0 Circle -7500403 true true 0 0 300 circle 2 false 0 Circle -7500403 true true 0 0 300 Circle -16777216 true false 30 30 240 cow false 0 Polygon -7500403 true true 200 193 197 249 179 249 177 196 166 187 140 189 93 191 78 179 72 211 49 209 48 181 37 149 25 120 25 89 45 72 103 84 179 75 198 76 252 64 272 81 293 103 285 121 255 121 242 118 224 167 Polygon -7500403 true true 73 210 86 251 62 249 48 208 Polygon -7500403 true true 25 114 16 195 9 204 23 213 25 200 39 123 cylinder false 0 Circle -7500403 true true 0 0 300 dot false 0 Circle -7500403 true true 90 90 120 face happy false 0 Circle -7500403 true true 8 8 285 Circle -16777216 true false 60 75 60 Circle -16777216 true false 180 75 60 Polygon -16777216 true false 150 255 90 239 62 213 47 191 67 179 90 203 109 218 150 225 192 218 210 203 227 181 251 194 236 217 212 240 face neutral false 0 Circle -7500403 true true 8 7 285 Circle -16777216 true false 60 75 60 Circle -16777216 true false 180 75 60 Rectangle -16777216 true false 60 195 240 225 face sad false 0 Circle -7500403 true true 8 8 285 Circle -16777216 true false 60 75 60 Circle -16777216 true false 180 75 60 Polygon -16777216 true false 150 168 90 184 62 210 47 232 67 244 90 220 109 205 150 198 192 205 210 220 227 242 251 229 236 206 212 183 fish false 0 Polygon -1 true false 44 131 21 87 15 86 0 120 15 150 0 180 13 214 20 212 45 166 Polygon -1 true false 135 195 119 235 95 218 76 210 46 204 60 165 Polygon -1 true false 75 45 83 77 71 103 86 114 166 78 135 60 Polygon -7500403 true true 30 136 151 77 226 81 280 119 292 146 292 160 287 170 270 195 195 210 151 212 30 166 Circle -16777216 true false 215 106 30 flag false 0 Rectangle -7500403 true true 60 15 75 300 Polygon -7500403 true true 90 150 270 90 90 30 Line -7500403 true 75 135 90 135 Line -7500403 true 75 45 90 45 flower false 0 Polygon -10899396 true false 135 120 165 165 180 210 180 240 150 300 165 300 195 240 195 195 165 135 Circle -7500403 true true 85 132 38 Circle -7500403 true true 130 147 38 Circle -7500403 true true 192 85 38 Circle -7500403 true true 85 40 38 Circle -7500403 true true 177 40 38 Circle -7500403 true true 177 132 38 Circle -7500403 true true 70 85 38 Circle -7500403 true true 130 25 38 Circle -7500403 true true 96 51 108 Circle -16777216 true false 113 68 74 Polygon -10899396 true false 189 233 219 188 249 173 279 188 234 218 Polygon -10899396 true false 180 255 150 210 105 210 75 240 135 240 house false 0 Rectangle -7500403 true true 45 120 255 285 Rectangle -16777216 true false 120 210 180 285 Polygon -7500403 true true 15 120 150 15 285 120 Line -16777216 false 30 120 270 120 leaf false 0 Polygon -7500403 true true 150 210 135 195 120 210 60 210 30 195 60 180 60 165 15 135 30 120 15 105 40 104 45 90 60 90 90 105 105 120 120 120 105 60 120 60 135 30 150 15 165 30 180 60 195 60 180 120 195 120 210 105 240 90 255 90 263 104 285 105 270 120 285 135 240 165 240 180 270 195 240 210 180 210 165 195 Polygon -7500403 true true 135 195 135 240 120 255 105 255 105 285 135 285 165 240 165 195 line true 0 Line -7500403 true 150 0 150 300 line half true 0 Line -7500403 true 150 0 150 150 pentagon false 0 Polygon -7500403 true true 150 15 15 120 60 285 240 285 285 120 person false 0 Circle -7500403 true true 110 5 80 Polygon -7500403 true true 105 90 120 195 90 285 105 300 135 300 150 225 165 300 195 300 210 285 180 195 195 90 Rectangle -7500403 true true 127 79 172 94 Polygon -7500403 true true 195 90 240 150 225 180 165 105 Polygon -7500403 true true 105 90 60 150 75 180 135 105 plant false 0 Rectangle -7500403 true true 135 90 165 300 Polygon -7500403 true true 135 255 90 210 45 195 75 255 135 285 Polygon -7500403 true true 165 255 210 210 255 195 225 255 165 285 Polygon -7500403 true true 135 180 90 135 45 120 75 180 135 210 Polygon -7500403 true true 165 180 165 210 225 180 255 120 210 135 Polygon -7500403 true true 135 105 90 60 45 45 75 105 135 135 Polygon -7500403 true true 165 105 165 135 225 105 255 45 210 60 Polygon -7500403 true true 135 90 120 45 150 15 180 45 165 90 square false 0 Rectangle -7500403 true true 30 30 270 270 square 2 false 0 Rectangle -7500403 true true 30 30 270 270 Rectangle -16777216 true false 60 60 240 240 star false 0 Polygon -7500403 true true 151 1 185 108 298 108 207 175 242 282 151 216 59 282 94 175 3 108 116 108 target false 0 Circle -7500403 true true 0 0 300 Circle -16777216 true false 30 30 240 Circle -7500403 true true 60 60 180 Circle -16777216 true false 90 90 120 Circle -7500403 true true 120 120 60 tree false 0 Circle -7500403 true true 118 3 94 Rectangle -6459832 true false 120 195 180 300 Circle -7500403 true true 65 21 108 Circle -7500403 true true 116 41 127 Circle -7500403 true true 45 90 120 Circle -7500403 true true 104 74 152 triangle false 0 Polygon -7500403 true true 150 30 15 255 285 255 triangle 2 false 0 Polygon -7500403 true true 150 30 15 255 285 255 Polygon -16777216 true false 151 99 225 223 75 224 truck false 0 Rectangle -7500403 true true 4 45 195 187 Polygon -7500403 true true 296 193 296 150 259 134 244 104 208 104 207 194 Rectangle -1 true false 195 60 195 105 Polygon -16777216 true false 238 112 252 141 219 141 218 112 Circle -16777216 true false 234 174 42 Rectangle -7500403 true true 181 185 214 194 Circle -16777216 true false 144 174 42 Circle -16777216 true false 24 174 42 Circle -7500403 false true 24 174 42 Circle -7500403 false true 144 174 42 Circle -7500403 false true 234 174 42 turtle true 0 Polygon -10899396 true false 215 204 240 233 246 254 228 266 215 252 193 210 Polygon -10899396 true false 195 90 225 75 245 75 260 89 269 108 261 124 240 105 225 105 210 105 Polygon -10899396 true false 105 90 75 75 55 75 40 89 31 108 39 124 60 105 75 105 90 105 Polygon -10899396 true false 132 85 134 64 107 51 108 17 150 2 192 18 192 52 169 65 172 87 Polygon -10899396 true false 85 204 60 233 54 254 72 266 85 252 107 210 Polygon -7500403 true true 119 75 179 75 209 101 224 135 220 225 175 261 128 261 81 224 74 135 88 99 wheel false 0 Circle -7500403 true true 3 3 294 Circle -16777216 true false 30 30 240 Line -7500403 true 150 285 150 15 Line -7500403 true 15 150 285 150 Circle -7500403 true true 120 120 60 Line -7500403 true 216 40 79 269 Line -7500403 true 40 84 269 221 Line -7500403 true 40 216 269 79 Line -7500403 true 84 40 221 269 x false 0 Polygon -7500403 true true 270 75 225 30 30 225 75 270 Polygon -7500403 true true 30 75 75 30 270 225 225 270 @#$#@#$#@ NetLogo 4.1 @#$#@#$#@ @#$#@#$#@ @#$#@#$#@ @#$#@#$#@ @#$#@#$#@ default 0.0 -0.2 0 0.0 1.0 0.0 1 1.0 0.0 0.2 0 0.0 1.0 link direction true 0 Line -7500403 true 150 150 90 180 Line -7500403 true 150 150 210 180 @#$#@#$#@ 0 @#$#@#$#@