# The Helmholtz Machine

Peter Dayan, Dept. of Computer Science, University of Toronto

Geoffrey E. Hinton, Dept. of Computer Science, University of Toronto

Radford M. Neal, Dept. of Computer Science, University of Toronto

Richard S. Zemel, The Salk Institute

Discovering the structure inherent in a set of patterns is a
fundamental aim of statistical inference or learning. One fruitful approach is to build a
parameterised stochastic generative model, independent draws from which are likely to
produce the patterns. For all but the simplest generative models, each pattern can be
generated in exponentially many ways. It is thus intractable to adjust the parameters to
maximize the probability of the observed patterns, We describe a way of finessing this
combinatorial explosion by maximising an easily computed lower bound on the probability of
the observations. Our method can be viewed as a form of hierarchical self-supervised
learning that may relate to the function of bottom-up and top-down cortical processing
pathways.

*In Neural Computation*, vol. 7, pp. 1022-1037 (1995).

Download: [ps.gz] [pdf]

**Associated reference:**
A stochastic algorithm related to the Helmholtz Machine is discussed in the
following paper: Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995) The
wake-sleep algorithm for unsupervised neural networks*, Science*, vol. 268, pp.
1158-1161. Download [abstract]
[ps] [pdf]

[home page] [publications]