Developing Population Codes by Minimizing Description
Length
Richard S. Zemel
University of Toronto and
The Salk Institute, CNL
La Jolla, CA
Geoffrey Hinton
Department of Computer Science
University of Toronto
Abstract
The Minimum Description Length principle (MDL) can be used to train
the hidden units of a neural network to extract a representation that is cheap to describe
but nonetheless allows the input to be reconstructed accurately. We show how
MDL can be used to develop highly redundant population codes. Each hidden unit has a
location in a low-dimensional implicit space. If the hidden unit activities
form a bump of a standard shape in this space, they can be cheaply encoded by the center
of this bump. So the weights from the input units to the hidden units in a
self-supervised network are trained to make the activities form a standard bump. The
coordinates of the hidden units in the implicit space are also learned, thus allowing
flexibility, as the network develops a discontinuous topography when presented with
different input classes. Population-coding in a space other than the input enables a
network to extract nonlinear higher-order properties of the inputs.
Download [ps] [pdf]
Advances in Neural Information Processing Systems 6. (1994)
J. D. Cowan, G. Tesauro and J. Alspector (Eds.), Morgan Kaufmann: San Mateo, CA.
[home page] [publications]