Lesson 9: Theory of Intelligence

Research on learning in deep networks has led to an impressive performance by machines on tasks such as object recognition, but a deep understanding of the behaviour of these networks and why they perform so well remains a mystery. In this unit, you will first learn about a model of rapid object recognition in the visual cortex that resembles the structure of deep networks. You will then learn some of the theory behind how the structural connectivity, complexity, and dynamics of deep networks govern their learning behaviour.

Tomaso Poggio describes a theory of processing in the ventral pathway of the brain that solves the challenging problem of recognizing objects despite variations in their visual appearance due to geometric transformations such as translation and rotation.

image

Tomaso Poggio and his colleagues have developed a model of the early processing stages in the ventral visual pathway of the brain, which may underlie our ability to recognize object categories from visual input in a brief flash of less than 100 milliseconds. (Courtesy of Tomaso Poggio and Thomas Serre. From “Models of visual cortex.” Scholarpedia 8 no. 4 (2013): 3516. License CC BY-NC-SA.)

The guest lecture by Surya Ganguli shows how insights from statistical mechanics applied to the analysis of high-dimensional data can contribute to our understanding of how functions such as object categorization emerge in multi-layer neural networks.

Haim Sompolinsky explores the theoretical role of common properties of neural architectures in biological systems, in learning tasks such as classification. These properties include the number of stages of the neural network, compression or expansion of the dimensionality of the information, the role of noise, and presence of recurrent or feedback connections.

Unit Activities

Useful Background

  • Introductions to statistics and machine learning, including deep learning networks
1 Like

Lesson 9.1: Visual Cortex & Deep Networks


Description: Describes the theoretical and empirical basis for i-theory, which embodies a hierarchical feedforward network model of processing in the ventral visual pathway of the primate brain, to support invariant object recognition.

Instructor: Tomaso Poggio


Click here for the lesson transcript

Click here for the lesson slides

Seminar 9: Statistical Physics of Deep Learning


Description: Describes how the application of methods from statistical physics to the analysis of high-dimensional data can provide theoretical insights into how deep neural networks can learn to perform functions such as object categorization.

Instructor: Surya Ganguli


Click here for the lesson transcript

Lesson 9.2: Sensory Representations in Deep Networks


Description: Analysis of common properties of sensory representations in deep cortex-like architectures, toward a systematic theoretical understanding of the capacity and limitations of deep learning networks.

Instructor: Haim Sompolinsky


Click here for the lesson transcript

Further Study

Additional information about the speakers’ research and publications can be found at their websites:

Advani, M., and S. Ganguli. This resource may not render correctly in a screen reader.“Statistical Mechanics of High-Dimensional Inference.” (PDF) (2016).

Anselmi, F., J. Z. Leibo, et al. “Unsupervised Learning of Invariant Representations.” Theoretical Computer Science 633 (2016): 112–21.

Babadi, B., and H. Sompolinsky. This resource may not render correctly in a screen reader.“Sparseness and Expansion in Sensory Representations.”(PDF - 2.8MB) Neuron 83 (2014): 1213–26.

Gao, P., and S. Ganguli. “On Simplicity and Complexity in the Brave New World of Large-Scale Neuroscience.” (PDF) Current Opinion in Neurobiology 32 (2015): 148–55.

Poggio, T. This resource may not render correctly in a screen reader.“Deep Learning: Mathematics and Neuroscience.” (PDF - 1.2MB) Center for Brains Minds & Machines Views & Reviews (2016).

Saxe, A., J. McClelland, et al. This resource may not render correctly in a screen reader.“Learning Hierarchical Category Structure in Deep Neural Networks.” (PDF) Proceedings 35th Annual Meeting of the Cognitive Science Society (2013): 1271–76.

Serre, T., G. Kreiman, et al.This resource may not render correctly in a screen reader. “A Quantitative Theory of Immediate Visual Recognition.” (PDF) Progress in Brain Research 165 (2007): 33–56.

Sompolinsky, H. “Computational Neuroscience: Beyond the Local Circuit.” (PDF) Current Opinion in Neurobiology 25 (2014): 1–6.