PALS0039 Introduction to Deep Learning for Speech and Language Processing
UCL Division of Psychology and Language Sciences

Week 7 - Language Modelling

In which we build models that predict the continuation of sentences and which can be used to generate text and improve the accuracy of speech recognition and machine translation.

Learning Objectives

By the end of the session the student will be able to:


  1. Probability of a sentence
  2. We discuss what it means to ascribe a probability to a sentence, and how that relates to conventional linguistic views of meaningfulness and grammaticality.

  3. Statistical language models
  4. We introduce a probabilistic model of text based on n-grams, describing how n-grams are used to estimate probabilities. We discuss problems with the n-gram approach, and one method for assessing the quality of a language model using perplexity.

  5. Neural language models
  6. We look at how recurrent networks can be used to build neural language models, and compare their performance with n-gram models on the same data. We demonstrate the training of a neural language model and compare it to an n-gram model trained on the same data.

  7. Applications of language models
  8. We look at how language models can be used to generate text, and how they can be used in machine translation and speech recognition to improve the quality of the output.

Research Paper of the Week


Be sure to read one or more of these discussions of language models:


Implement answers to the problems described in the notebooks below. Save your completed notebooks into your personal Google Drive account.

    1. N-gram language modelling using NLTK
    2. Language modelling using RNNs

Word count: . Last modified: 22:45 11-Mar-2022.