Peter Dayan, Dept. of Brain and Cognitive Sciences, Massachusetts Institute of Technology

We describe a linear network that models correlations between
real-valued visible variables using one or more real-valued hidden
variables - a *factor analysis* model. This model can be seen
as a linear version of the ``Helmholtz machine'', and its parameters
can be learned using the ``wake-sleep'' method, in which learning of
the primary ``generative'' model is assisted by a ``recognition''
model, whose role is to fill in the values of hidden variables based
on the values of visible variables. The generative and recognition
models are jointly learned in ``wake'' and ``sleep'' phases, using just
the delta rule. This learning procedure is comparable in simplicity to
Oja's version of Hebbian learning, which produces a somewhat different
representation of correlations in terms of principal components. We
argue that the simplicity of wake-sleep learning makes factor analysis
a plausible alternative to Hebbian learning as a model of
activity-dependent cortical plasticity.

Technical Report No. 9607, Dept. of Statistics, University of Toronto (July 1996), 23 pages: postscript, pdf, associated software.

There is software available on-line that implements the method described.

Neal, R. M. and Dayan, P. (1997) ``Factor analysis using delta-rule wake-sleep learning'',Neural Computation, vol. 9, pp. 1781-1803: abstract, associated references, associated software.

The ``wake-sleep'' algorithm is described in the following paper:

Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995) ``The ``wake-sleep'' algorithm for unsupervised neural networks'',The wake-sleep algorithm is a way of learning ``Helmholtz Machines'', which are discussed in the following paper:Science, vol. 268, pp. 1158-1161: abstract, associated references.

Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995) ``The Helmholtz machine'',Neural Computation, vol. 7, pp. 1022-1037: abstract, associated reference.