The Role of Constraints in Hebbian Learning

Kenneth D. Miller and David J.C. MacKay

Models of unsupervised correlation-based (Hebbian) synaptic plasticity are typically unstable: either all synapses grow until each reaches the maximum allowed strength, or all synapses decay to zero strength. A common method of avoiding these outcomes is to use a constraint that conserves or limits the total synaptic strength over a cell. We study the dynamical effects of such constraints. Two methods of enforcing a constraint are distinguished, multiplicative and subtractive. For otherwise linear learning rules, multiplicative enforcement of a constraint results in dynamics that converge to the principal eigenvector of the operator determining unconstrained synaptic development. Subtractive enforcement, in contrast, leads to a final state in which almost all synaptic strengths reach either the maximum or minimum allowed value. This final state may be dominated by weight configurations other than the principal eigenvector of the unconstrained operator. Thus, multiplicative constraints yield a ``graded" receptive field in which all mutually correlated inputs typically retain some representation, whereas subtractive constraints yield a receptive field that is ``sharpened" to a few maximally-correlated inputs. If two equivalent input populations (e.g. two eyes) innervate a common target, multiplicative constraints prevent their segregation (ocular dominance segregation) when the two populations are weakly correlated; whereas subtractive constraints allow segregation under these circumstances. An approach to understanding constraints over input and over output cells is suggested, and some biological implementations are discussed.

postscript (UCSF,USA) and postscript (Cambridge,UK) as appeared in Neural Computation (1994). technical report version (USA) and technical report version (UK) (Longer, with a few minor corrections)

@ARTICLE{MM94:nc,
 KEY            ="",
 AUTHOR         ="K. D. Miller and D. J. C.  MacKay",
 TITLE          ="The role of constraints in {H}ebbian learning",
 JOURNAL        ="Neural Computation",
 VOLUME		="6",
 NUMBER 	="1",
 PAGES          ="98-124",
 YEAR           ="1994",
 ANNOTE ="Date submitted: 9 Oct 1992; Date accepted: 13 May 1993; 
                  Collaborating institutes:
		  California Institute of Technology"}

David MacKay's: home page, publications, BibTeX file.