Re: Competitive networks with recurrent connections

["Randall C. O'Reilly" <oreilly@grey.colorado.edu>]

Dennis Garlick <dennisg@psych.usyd.edu.au> writes:
> I'm interested in trying competitive networks that have recurrent 
> connections.

There are lots of issues that come up in incorporating settling
dynamics (needed for recurrent connections) with soft competitive
hebbian learning.  cs++, which does settling, does have a simpler form
of Hebb learning that is compatible with weights that are both + and -.
SCL requires weights to match activations and it gets tricky --
everything works best when both weights and activations are 0-1, but
then you need some kind of inhibition, and the usual hard WTA won't
be of much use because you need smoothly graded activations to do
settling.. I suppose a softmax might do something useful, but this
isn't implemented anywhere in PDP++ -- it might be pretty easy to add
to cs++ though.

You can also try leabra++ and set the lmix.hebb value to 1 -- this is
then a settling network that allows recurrent connections and is using
the soft competitive learning rule.  It also has a KWTA activation
function and the unit activations are computed as point neurons -- you
can set k to 1 to get one unit active per layer, or set k higher and
get useful distributed representations..

				- Randy