Bayesian Learning, 7.5 credits

---

Contents

The course gives a gentle but solid introduction to Bayesian statistical inference, with special emphasis on models and methods in computational statistics and machine learning.

• We will get off to a shocking start by introducing a very different probability concept than the one you are probably used to: subjective probability.
• We will then move on to the mathematics of the prior-to-posterior updating in basic statistical models, such as the Bernoulli, normal and multinomial models.
• Bayesian prediction and decision making under uncertainty is carefully explained, and you will hopefully see why Bayesian methods are so useful in modern application where so much focuses on prediction and decision making.
• The really interesting stuff starts to happen when we study regression and classification. You will learn how prior distributions can be seen as regularization that allows us to use flexible models with many parameters without overfitting the data and improving predictive performance.
• A new world will open up when we learn how complex models can be analyzed with simulation methods like Markov Chain Monte Carlo (MCMC), Hamiltonian Monte Carlo (HMC) and approximate optimization methods like Variational Inference (VI).
• You will also get a taste for probabilistic programming languages for Bayesian learning, in particular the popular Stan language in R.
• Finally, we'll round off with an introduction to Bayesian model selection and Bayesian variable selection.

---

Course literature and Schedule

The course will use the following book as the main course literature:

• Gelman, Carlin, Stern, Dunson, Vehtari, Rubin (2014). Bayesian Data Analysis (BDA). Chapman & Hall/CRC: Boca Raton, Florida. 3rd edition. Here is the book webpage and a free PDF version.
• Villani, M. (2024). Bayesian Learning (BL). Little more than half of a book that I am writing is here. This is work in progress, but currently covers L1-L7 and L11.
• Additional course material linked from this page, such as articles and tutorial.

The course schedule on TimeEdit is here: Schedule.

Some of the Lectures are pre-recorded (see the Schedule on TimeEdit), partly to make room for the discussion seminars. The recorded lectures are hosted on Athena in a folder called "Videos". See the file videos in the Resources at Athena for a lists of the videos, and the order in which they should be viewed.

---

Computer labs

• The three computer labs are central to the course. Expect to allocate substantial time for each lab, not just the scheduled time for computer exercises. Many of the exam questions will be computer based, so working on the labs will also help you with the exam.

• The labs should be done in R, since only R is available at the computer exam and your lab solutions will be available for you at the exam.

• The labs should be done in pairs of students.

• Each lab report should be submitted as a PDF along with the .R file with code. Submission is done through Athena.

• There is only two hours of supervised time allocated to each lab. The idea is that you:

  • should start working on the lab before the computer session
  • so that you are in a position to ask questions at the session
  • and then finish up the report after the lab session.

• The datasets used in the labs are included in the SUdatasets R package which is available on github. There is also a course package bayeslearn with some distributions and functions for regression, logistic regression etc. To install the packages, you first need to install the remotes package.

# To install and load the package with the data
install.packages("remotes") # only one time
library(remotes)  
install_github("StatisticsSU/SUdatasets") # only one time
library(SUdatasets)
install_github("StatisticsSU/bayeslearn") # only one time
library(bayeslearn)

# For a list of all available data sets
help(, "SUdatasets")

# For information about specific data sets
help(name_of_dataset)

---

Interactivity

I have created a number of interactive widgets that can be used to explore the concepts in the course. The widgets are created in Observable, a platform for creating interactive web-based documents. The widgets are linked from the course schedule, but can also be found in:

• Bayesian Learning collection.
• The Distributions collection.
• All collections (for different courses and topics) can be found here.

---

Teachers

Mattias Villani, Lecturer
Professor, Natural Born Bayesian 😜
Department of Statistics, Stockholm University

Oscar Oelrich, Math exercises and Computer labs
PhD in Statistics, specializing in Bayesian Statistics
Department of Statistics, Stockholm University

---

Lectures

Lecturer: Mattias Villani

Lecture 1 - Subjective probability. Bayesian updating. Bernoulli model. Gaussian model.
Reading: BDA Ch. 1, 2.1-2.4 | BL Ch. 1-2, 4 | Slides
Notebooks: Analyzing email spam data with a Bernoulli model | Analyzing internet download speeds with a Gaussian model
Widgets: Bernoulli data - beta prior | Gaussian known variance

Lecture 2 - Poisson model. Summarizing posteriors. Conjugate priors. Prior elicitation.
Reading: BDA Ch. 2.-2.9 | BL Ch. 2 and 4 | Slides
Notebooks: Analyzing number of eBay bidders with an iid Poisson model
Widgets: Poisson data - Gamma prior | Exponential data - Gamma prior | Credible intervals | Autoregressive processes | Second derivative measures curvature

Lecture 3 - Multi-parameter models. Marginalization. Monte Carlo simulation. Multinomial model.
Reading: BDA Ch. 3 | BL Ch. 3 | Slides
Notebooks: Analyzing mobile phone survey data with a multinomial model
Widgets: Scaled inverse chi2 distribution | Multinomial data - Dirichlet prior

Lecture 4 - Linear Regression. Prediction. Making Decisions. Reading: BDA Ch. 14, Ch. 9.1-9.2 | BL Ch. 5 and 6 | Slides
Code: Prediction with two-parameter Gaussian model
Widgets: linear regression | prediction iid Gaussian

Lecture 5 - Classification. Large-sample properties and Posterior approximation.
Reading: BDA Ch. 16.1-16.3, 4.1-4.2 | BL 7-8 and appendix on Taylor approximations | Slides
Notebook: Logistic regression
Code: bayeslearn package - Logistic and Probit regression for Titanic data
Widgets: posterior approximation Beta model
Extras: Video on Taylor approximation

Lecture 6 - Nonlinear regression and Classification. Regularization.
Reading: BDA Ch. 20.1-20.2 | BL 12.1-12.4 | Slides

Lecture 7 - Gibbs sampling and Data augmentation
Reading: BDA Ch. 10-11 | BL 9.1-9.2 and 13.1-13.3 | Slides
Code: Gibbs sampling for a bivariate normal | Gibbs sampling for a mixture of normals
Widget: mixture of normals
Extras: Illustration of Gibbs sampling when parameters are strongly correlated.

Lecture 8 - MCMC and Metropolis-Hastings
Reading: BDA Ch. 11 | Slides
Widget: Random walk Metropolis
Code: Simulating Markov Chains

Lecture 9 - HMC and Variational inference
Reading: BDA Ch. 12.4 and Ch. 13.7 | Slides
Widget: Hamiltonian dynamics and leapfrog | HMC from multimodal distribution
Extras: HMC animations | HarlMCMC shake | visual explantion of HMC algorithm tuning | Radford Neal's handbook chapter on HMC

Lecture 10 - Probabilistic programming for Bayesian learning
Reading: RStan vignette | Linear regression in RStan | Prediction in RStan | Slides
RStan Code: Bernoulli | Normal model | Logistic regression | Logistic regression with random effects | Poisson model | Three plants data | Regularized L2 regression | fossil data for regularized regression
Turing code (Julia): Bernoulli model | Normal model

Lecture 11 - Bayesian model comparison
Reading: BDA Ch. 7 | BL Ch. 12 | Slides
Code: Comparing models for count data

Lecture 12 - Bayesian variable selection and model averaging
Reading: Article on Bayesian variable selection | Slides

---

Mathematical exercises

Assistant: Oscar Oelrich

Exercise session 1: Problem set

Exercise session 2: Problem set

---

Computer labs

Assistant: Oscar Oelrich

Computer Lab 1 - Exploring posterior distributions in one-parameter models by simulation and direct numerical evaluation.
Lab: Lab 1
Submission: Athena.

Computer Lab 2 - Polynomial regression and classification with logistic regression.
Lab: Lab 2
Submission: Athena.

Computer Lab 3 - Gibbs sampling, Metropolis-Hastings and Stan.
Lab: Lab 3 | Note on how to code R functions which take functions as arguments
Submission: Athena.

---

Examination

The course examination consists of:

• Written lab reports (deadlines given in Athena)
• Exam (pen and paper + computer)

The written exam will be in a room with computer access and R will be installed. Roughly 1/4 of the exam will be pen-and-paper math problems, and 3/4 computer based problems. You will be given the opportunity to submit your lab solutions as a PDF so that they will be available to you during the exam. The uploaded lab solutions can be polished, but are only allowed to contain material related to the lab solutions. The uploaded lab solutions will be screened before the exam. You will also have the following PDF with distributions available during the exam Distributions from BDA3.

Some old exams with solutions:

• Old exams

---

R

• The main page with links to downloads for the programming language R
• RStudio - a very nice developing environment for R.
• An introduction to R | John Cook's intro to R for programmers | base-R cheat sheet .