[pdp-discuss] TD demo and delayed rewards

[Randall C. O'Reilly]

I'm not sure there is a simple solution to this problem.  It is clear that in 
the brain delayed rewards ("trace" conditioning) depends on prefrontal cortex 
+ hippocampus;  If you want a reasonable model of these systems, it is going 
to be a bit more complex than a simple demo.. 

Our recent "PBWM" (prefrontal cortex basal ganglia working memory) model 
(O'Reilly & Frank, 2006 -- avail on my Online Papers on my webpage) is the 
latest incarnation of our thinking about how the PFC solves this problem in 
conjunction with TD-like learning..  Also see the in press PVLV paper for 
more details on the TD-like part of it..

- Randy

On Thursday 01 February 2007 11:35, allan.randall at ntt.ca wrote:
> The simple TD demo in "Explorations" (p 199) is a really nice way of
> showing the basic algotithm at work. Even the serial compound "cheat" I
> find is readily accepted when I explain the demo to others. The bigger
> problem is the fact that the stimulus must be maintained up to the point of
> reward... this is readily apparent when you show the demo, and makes it
> look like it is not really detecting delayed reward at all.
>
> What would be the easiest way to fix this, without destroying the simple,
> pedagogically clean presentation of the demo? The active memory model (p
> 307) gets into numerous other things, and is no longer just a demo of the
> basic TD algorithm.
>
> I'm guessing the problem could be fixed with a simpler modification of the
> basic TD demo... but maybe that will get me into trouble? Anyone try this?
>
> I'm not questioning the value of the algorithm... I understand why the demo
> makes the compromise it does. I'm just trying to respond to objections that
> I have heard raised by those watching the demo, without getting into a
> whole other more complicated demo.
>
> Any thoughts would be appreciated.
>
> Cheers,
>
> Allan Randall, NTT Systems Inc.
>
> _______________________________________________
> PDP-Discuss mailing list
> PDP-Discuss at psych.Colorado.EDU
> http://psych.colorado.edu/mailman/listinfo/pdp-discuss