PALS0039 Introduction to Deep Learning for Speech and Language Processing
UCL Division of Psychology and Language Sciences

Week 3 - Artificial Neural Networks

In which we look at the general structure of artificial neural networks, including the mathematical description of processing within a node, and how networks can learn by gradient descent.

Learning Objectives

By the end of the session the student will be able to:


  1. Neural networks for machine learning
  2. We view the brain as an information processing system, looking at the operation of neurons and how a network of neurons can perform complex calculations. We discuss how artificial neural networks can be motivated by the processing networks of the brain without being simulations of biological neurons.

  3. The Perceptron
  4. We discuss a simple mathematical of a neuron proposed by McCulloch and Pitts, and a means to train such a model from data proposed by Rosenblatt: the Perceptron learning rule.

  5. Multiple layers of perceptrons
  6. We look at the criticisms of perceptrons as a means to perform information processing, and a solution to the problem of training multiple layers of perceptrons through the use of gradient descent.

  7. Deep neural networks
  8. We briefly review the history of how networks of multi-layer perceptrons turned into "deep" networks. Problems in extending gradient descent to large networks were gradually overcome by improvements in algorithms and an increase in computer power. We outline common activation functions, loss functions and optimisation methods.

  9. Programing DNNs
  10. We introduce the Keras toolkit for building, training and running deep neural networks.

Research Paper of the Week

Web Resources


Be sure to read one or more of these discussions of deep learning:


Implement answers to the problems described in the notebooks below. Save your completed notebooks into your personal Google Drive account.

    1. Keras practice
    2. Classification with a neural network

Word count: . Last modified: 22:45 11-Mar-2022.