[PDP-discuss] Higher error in test than train?

[Randall C. O'Reilly]

Very strange.. I wonder if the network might be learning some sequential 
dependency in the input -- are you using permuted order or sequential order 
of events?  You might want to look at the trial-by-trial behavior just after 
switching off learning -- does it deteriorate immediately or more slowly over 
trials?  Also try setting the lrate = 0 in the conspec instead of using 
test -- if that gives the same bad results, then you DEFINITELY have a case 
of the network learning in some way that significantly improves performance..  
These networks can be mighty crafty sometimes.. ;)

- Randy

On Monday 25 June 2007 04:28, Frank Leoné wrote:
> Hi mister O'Reilly,
>
> Thanks for the fast reply. I checked all the variables, but didn't find any
> differences. On the contrary: if I use the same processes for training and
> testing and in the sequence epoch just switch "online" to "test", the same
> high error arrises. If I then switch it back to online, the error is low
> again. The nasty thing is, not only the error is higher, but also the
> actual output it worse: so it is not that the number is incorrect, the
> network actually gives a different output. Both the SE-stat as the
> errorscript give far worse values, consistent with worse output.
>
> The network is made up of scalarvallayers, with one hidden layer with a low
> decay and sequences of trials. Nothing really special. If I train the
> network without multiple events in the groups, so without any need of
> memory, the problem stays.
>
> I'll try to figure out what the problem is, hopefully you got some helpful
> hints.
>
> Thanks in advance,
>
> kind regards,
>
>      Frank
>
> _________________________________________________________________
> Talk with your online friends with Messenger
> http://www.join.msn.com/messenger/overview