Hi
I am implementing bits of Leabra using visual basic and excel and comparing this to the simulations under PDP++ to try and verify to myself that I understand what’s going on. I have succeeded in doing this without any problems for the simple hebbian PCA and the basic CPCA (computational explorations pp 122-132). However I get a problem when I add in the equations for renormalisation and contrast enhancement (pp132- 137) and I am not able to reproduce all of the results obtained in PDP++. Using the three lines environment in the hebb_correl.proj.gz I get similar results with values of savg_cor=1, wt_gain =1, wt_off=1 and savg_cor=1, wt_gain =6, wt_off=1. When I increase wt_off rather than being left with strong weights just for the five units in the central cross, the right leaning diagonals become more strongly weighted and the left leaning ones also increase their weights. Decreasing wt_off does lead to a situation where just the central cross has strong biases, however further decreases lead to the strong biases switching to weak ones – but not in a graded way as in PDP++. Decreasing savg_cor has the same effect as in PDP++ up to a point – reducing to 0.7 (wt_gain=6, wt_off=1) leaves the central cross with stronger weights, however further reductions in savg_cor causes the two non-central right diagonals to become strongest – and the middle diagonal unweighted, and reducing to 0 causes the one of the right leaning diagonals (it varies from simulation to simulation) to become most weighted and the diagonals either side disappear (in PDP++ all the weights reduce to 0 when you do this).
As far as I can ascertain I have implemented the equations exactly as given in the book:
I have computed the expected activity by averaging over all the inputs and over all the pattern presentations:
alpha(t) = (sum over t of (sum over i of input unit activation levels))/(i*t)
Where i is the number of sending units and t is the number of pattern presentations
I compute m as indicated in the book, and apply the constraint that a weight cannot exceed this value. I use the weight update equation (4.18) on the linear weights (converted from effective weights – equation 4.23 in reverse) and convert the new weights back to the effective weights (equation 4.23) to update the receiving unit activation (using equation 4.1). I clip the effective weights and the linear weights to the 0 to 1 range (actually 0.0005 to .9995 to avoid dividing by zero). Is there something else I need to do in order to get the right behaviour?
Below is what I get in my excel/visual basic simulation:
For all the below:
3 line' input, probability patterns to r +.2333, to l = .1, lrate=0.005. 25 sending unit and 1 receiving unit as in hebb_correl.proj.gz
20 epochs each with 100 pattern presentations (2000 cycles in all)
manipulation of wt_off:
savg_cor = 1 wt_gain = 6 wt_off = 1
Weights to receiving unit 1
0.000588824 0.017397858 1.5672E-20 0.987872661 0.986706461
0.006005861 0.000588824 0.9995 0.986706461 0.988948449
1.5672E-20 0.99934584 0.999254323 0.9995 1.5672E-20
0.987872661 0.986706461 0.999427419 0.000588824 0.017397858
0.986706461 0.988948449 1.5672E-20 0.006005861 0.000588824
savg_cor = 1 wt_gain = 6 wt_off = 1.25
Weights to receiving unit 1
0.031775943 0.004009276 5.97838E-20 0.997706442 0.997474366
0.007847263 0.031775943 0.9995 0.997474366 0.997919482
5.97838E-20 0.999192674 0.999439289 0.9995 5.97838E-20
0.997706442 0.997474366 0.999276493 0.031775943 0.004009276
0.997474366 0.997919482 5.97838E-20 0.007847263 0.031775943
savg_cor = 1 wt_gain = 6 wt_off = 2.5
Weights to receiving unit 1
0.683709288 0.39641841 3.82616E-18 0.999418964 0.999375921
0.453509842 0.683709288 0.999422301 0.999375921 0.99945931
3.82616E-18 0.999421154 0.9995 0.999462415 3.82616E-18
0.999418964 0.999375921 0.999461348 0.683709288 0.39641841
0.999375921 0.99945931 3.82616E-18 0.453509842 0.683709288
savg_cor = 1 wt_gain = 6 wt_off = 0.8
Weights to receiving unit 1
0.002229202 0.002107378 4.10831E-21 0.920930697 0.914358434
0.002019125 0.002229202 0.9995 0.914358434 0.927095004
4.10831E-21 0.999141593 0.999412288 0.9995 4.10831E-21
0.920930697 0.914358434 0.999263723 0.002229202 0.002107378
0.914358434 0.927095004 4.10831E-21 0.002019125 0.002229202
savg_cor = 1 wt_gain = 6 wt_off = 0.5
Weights to receiving unit 1
5.53534E-06 0.000231765 2.44874E-22 0.622586345 0.691880607
4.78365E-05 5.53534E-06 0.998999217 0.691880607 0.645050079
2.44874E-22 0.999361196 0.996204996 0.999199741 2.44874E-22
0.622586345 0.691880607 0.999364882 5.53534E-06 0.000231765
0.691880607 0.645050079 2.44874E-22 4.78365E-05 5.53534E-06
savg_cor = 1 wt_gain = 6 wt_off = 0.25
Weights to receiving unit 1
1.35398E-06 2.71898E-06 3.82616E-24 0.012007794 0.016112128
8.65354E-07 1.35398E-06 0.998825327 0.016112128 0.013110199
3.82616E-24 0.99801198 0.996448501 0.998930727 3.82616E-24
0.012007794 0.016112128 0.9985978 1.35398E-06 2.71898E-06
0.016112128 0.013110199 3.82616E-24 8.65354E-07 1.35398E-06
savg_cor = 1 wt_gain = 6 wt_off = 0.1
Weights to receiving unit 1
2.77117E-07 1.85564E-08 1.5672E-26 0.077573883 0.062786887
3.91346E-08 2.77117E-07 0.998610673 0.062786887 1.64806E-26
1.5672E-26 0.998629875 0.9995 1.86015E-08 1.5672E-26
0.077573883 0.062786887 3.92262E-08 2.77117E-07 1.85564E-08
0.062786887 1.64806E-26 1.5672E-26 3.91346E-08 2.77117E-07
this does the opposite - more or less-
of the simulations in computational
explorations pp136-137
manipulation of savg_cor:
savg_cor = 1 wt_gain = 6 wt_off = 1
Weights to receiving unit 1
0.003254032 0.004562992 1.5672E-20 0.990709185 0.989783909
0.001926414 0.003254032 0.999406105 0.989783909 0.991560243
1.5672E-20 0.999427419 0.9993219 0.999481061 1.5672E-20
0.990709185 0.989783909 0.9995 0.003254032 0.004562992
0.989783909 0.991560243 1.5672E-20 0.001926414 0.003254032
savg_cor = 0.7 wt_gain = 6 wt_off = 1
Weights to receiving unit 1
0.00018678 8.94641E-05 1.5672E-20 0.16193097 0.154859934
0.000100933 0.00018678 0.930540638 0.154859934 0.169288772
1.5672E-20 0.935137647 0.952844802 0.934002356 1.5672E-20
0.16193097 0.154859934 0.938393905 0.00018678 8.94641E-05
0.154859934 0.169288772 1.5672E-20 0.000100933 0.00018678
savg_cor = 0.6 wt_gain = 6 wt_off = 1
Weights to receiving unit 1
8.01611E-20 0.000822376 1.5672E-20 0.86465253 9.97379E-17
0.001484033 8.01611E-20 0.999379547 9.97379E-17 0.872801855
1.5672E-20 0.999427419 2.31872E-16 0.999436657 1.5672E-20
0.86465253 9.97379E-17 0.9995 8.01611E-20 0.000822376
9.97379E-17 0.872801855 1.5672E-20 0.001484033 8.01611E-20
savg_cor = .4 wt_gain = 6 wt_off = 1
Weights to receiving unit 1
2.39916E-20 0.000732184 1.5672E-20 0.316758841 3.5295E-19
0.000137333 2.39916E-20 0.99356707 3.5295E-19 0.329156147
1.5672E-20 0.977774506 5.46013E-19 0.993982648 1.5672E-20
0.316758841 3.5295E-19 0.979090731 2.39916E-20 0.000732184
3.5295E-19 0.329156147 1.5672E-20 0.000137333 2.39916E-20
savg_cor = .1 wt_gain = 6 wt_off = 1
Weights to receiving unit 1
5.94061E-19 5.14156E-05 3.64969E-19 0.040862513 2.76782E-18
6.42463E-05 5.94061E-19 0.684741738 2.76782E-18 0.040637454
3.64969E-19 0.71065113 3.93476E-18 0.68357516 3.64969E-19
0.040862513 2.76782E-18 0.709538175 5.94061E-19 5.14156E-05
2.76782E-18 0.040637454 3.64969E-19 6.42463E-05 5.94061E-19
savg_cor = 0 wt_gain = 6 wt_off = 1
Weights to receiving unit 1
0.00448853 2.59026E-10 4.99186E-11 5.12288E-09 0.982315167
3.88528E-10 0.00448853 1.2E-08 0.982315167 1.94551E-09
4.99186E-11 1.50559E-08 0.9995 5.172E-09 4.99186E-11
5.12288E-09 0.982315167 6.68931E-09 0.00448853 2.59026E-10
0.982315167 1.94551E-09 4.99186E-11 3.88528E-10 0.00448853
whereas the weights in comp explorations fade out with reducing savg_cor, mine don't
manipulation of wt_gain:
savg_cor = 1 wt_gain = 1 wt_off = 1
Weights to receiving unit 1
0.286860595 0.323904622 0.0005 0.668777778 0.665404626
0.265484345 0.286860595 0.986471307 0.665404626 0.672167951
0.0005 0.931789387 0.951833636 0.990856229 0.0005
0.668777778 0.665404626 0.935608206 0.286860595 0.323904622
0.665404626 0.672167951 0.0005 0.265484345 0.286860595
savg_cor = 1 wt_gain = 4 wt_off = 1
Weights to receiving unit 1
0.041521986 0.017146175 6.26252E-14 0.960898264 0.958386229
0.01262613 0.041521986 0.999473517 0.958386229 0.952475191
6.26252E-14 0.999179043 0.999359477 0.999255917 6.26252E-14
0.960898264 0.958386229 0.998867314 0.041521986 0.017146175
0.958386229 0.952475191 6.26252E-14 0.01262613 0.041521986
savg_cor = 1 wt_gain = 10 wt_off = 1
Weights to receiving unit 1
0.000453148 1.13804E-05 9.81459E-34 0.999131532 0.999475157
0.000203361 0.000453148 0.999317519 0.999475157 0.999255
9.81459E-34 0.999317519 0.9995 0.999415358 9.81459E-34
0.999131532 0.999475157 0.999415358 0.000453148 1.13804E-05
0.999475157 0.999255 9.81459E-34 0.000203361 0.000453148