Warning regarding the training of hybrid networks on sequences

[James Hammerton <james@tardis.ed.ac.uk>]

Hi,

This will be of general interest, as it involves a subtle mistake that
one can easily make in PDP++ (and which the manual gives NO warning
about). I've just discovered that several months worth of training
runs have been done incorrectly due to the fact that a ForkProcess
does not handle the housekeeping associated with the second
network. Whilst I was aware of the need to add a script to
reinitialise the weights at the start of each training run, I didn't
realise until today when trying to work out why a network seemed to
work perfectly when being run through a training process but not when
being driven by a script, that you also need to ensure that the
InitState function in the second network is run at the start of each
sequence, since the processes will only do this for the first
network. As a consequence of this my training of a hybrid network
involving a SOM for sequences and a SRN was performed without the
context layer being reset at all during training. In effect the SRN
was being run over a continuous stream of input, formed by the
concatenation of the training sequences being repeated each epoch!

The moral is if you are training hybrid networks, make doubly sure
that all the relevant housekeeping is being done for the second
network in your training processes. Having looked at the manual there
is no mention of the need to do this in the chapter on processes. I
strongly recommend that this omission be rectified, as it is an easy
mistake to make, a difficult error to track down and it could cause
real problems for people's work. 

James