Research on learning in deep networks has led to an impressive performance by machines on tasks such as object recognition, but a deep understanding of the behaviour of these networks and why they perform so well remains a mystery. In this unit, you will first learn about a model of rapid object recognition in the visual cortex that resembles the structure of deep networks. You will then learn some of the theory behind how the structural connectivity, complexity, and dynamics of deep networks govern their learning behaviour.
Tomaso Poggio describes a theory of processing in the ventral pathway of the brain that solves the challenging problem of recognizing objects despite variations in their visual appearance due to geometric transformations such as translation and rotation.
Tomaso Poggio and his colleagues have developed a model of the early processing stages in the ventral visual pathway of the brain, which may underlie our ability to recognize object categories from visual input in a brief flash of less than 100 milliseconds. (Courtesy of Tomaso Poggio and Thomas Serre. From “Models of visual cortex.” Scholarpedia 8 no. 4 (2013): 3516. License CC BY-NC-SA.)
The guest lecture by Surya Ganguli shows how insights from statistical mechanics applied to the analysis of high-dimensional data can contribute to our understanding of how functions such as object categorization emerge in multi-layer neural networks.
Haim Sompolinsky explores the theoretical role of common properties of neural architectures in biological systems, in learning tasks such as classification. These properties include the number of stages of the neural network, compression or expansion of the dimensionality of the information, the role of noise, and presence of recurrent or feedback connections.
Unit Activities
Useful Background
Introductions to statistics and machine learning, including deep learning networks
Description: Describes the theoretical and empirical basis for i-theory, which embodies a hierarchical feedforward network model of processing in the ventral visual pathway of the primate brain, to support invariant object recognition.
Description: Describes how the application of methods from statistical physics to the analysis of high-dimensional data can provide theoretical insights into how deep neural networks can learn to perform functions such as object categorization.
Lesson 9.2: Sensory Representations in Deep Networks
Description: Analysis of common properties of sensory representations in deep cortex-like architectures, toward a systematic theoretical understanding of the capacity and limitations of deep learning networks.