Deep and Probabilistic Machine Learning

This research meeting covers areas at the interface between deep learning and probabilistic learning, though individual topics can be in one or the other or both areas. The meetings are currently held online via Microsoft Teams or Zoom. Email Yee Whye if you are interested in attending but not in the group. We meet on a weekly basis on Wednesdays 1130-1200 and 1600-1700.

Organisers: Yee Whye Teh

2022

Elevenese Research Talk

Date	Presentation	Speakers/notes
6/4/2022	Implicit regularization in (stochastic) gradient descent	Alex Buna-Marginean
30/3/2022	Diffusions and Deep BSDE	Chris WIlliams (recording, passcode: bnF0!@!*)
23/3/2022	Latent SDEs and Infinitely Deep BNNs	Jin Xu (slides)
16/3/2022	Strange Behaviour of Deep Generative Models	Andrew Campbell
9/3/2022	Denoising Diffusion Models on Arbitrary State Spaces	Joe Benton (slides)
2/3/2022	Bootstrap your own latent	Mrinank Sharma
23/2/2022	FFCV	Nic Fishman(slides)
16/2/2022	Normalizing flows on manifolds	Emile Mathieu (slides)

Afternoon Research Talk

Date	Presentation	Speakers/notes
20/4/2022	Tuning GPT-3 on a Single GPU via Zero-Shot Hyperparameter Transfer	Greg Yang (slides and recording, Passcode: zT#?Nsa1)
6/4/2022	PDO-eConvs: Partial Differential Operator Based Equivariant Convolutions and Steerable Partial Differential Operators for Equivariant Neural Networks	Jin Xu and Ning Miao (slides)
30/3/2022	Learning Differential Equations that are Easy to Solve	Desi Ivanova
23/3/2022	Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data	Spencer Frei
16/3/2022	Wide Mean-Field Bayesian Neural Networks Ignore the Data	Mrinank Sharma
9/3/2022	On the Role of Neural Collapse in Transfer Learning	Tomer Galanti
2/3/2022	DiffusionNet and Graph Neural Diffusion	Michael Hutchinson and Emile Mathieu(recording, passcode: DRdDcn1^)
23/2/2022	Neural Collapse	Bobby He and Bryn Elesedy(slides)
16/2/2022	BAX, InfoBAX, and Applications to Experimental Design and Reinforcement Learning	Willie Neiswanger (Stanford) (recording, password: 4MX64%Ca)
9/2/2022	What Are Bayesian Neural Network Posteriors Really Like?	Yee Whye Teh

2021

Date	Presentation	Speakers/notes
1/12/2021	Representation Learning for Reinforcement Learning	Marc Bellemare (Google Brain Montreal, MILA), Clare Lyle (OatML) and Charline le Lan (OxCSML) (recording, password: ^Lv^1+Qh))
24/11/2021	Group Equivariant Subsampling Frame Averaging for Invariant and Equivariant Network Design Training independent subnetworks for robust prediction	Jin (notes) and Ning(notes)
10/11/2021	On Optimal Interpolation in Linear Regression Uniform Sampling over Episode Difficulty	Eduard and Guneet
13/10/2021	Small Data, Big Decisions: Model Selection in the Small-Data Regime Prequential MDL for Causal Structure Learning with Neural Networks	Cong and Andrew (notes)
21/4/2021	Intro to implicit function representations and point clouds & sets	Emilien
7/4/2021, 14/4/2021	Easter break
31/3/20201	RL Theme: SAC and TRPO	Cong and Andrew
24/3/2021	RL Theme: DQN	Desi
17/3/2021	RL Theme: viewing party	Simons Institute Control Fundamentals
10/3/2021	RL Theme: viewing party	Simons Institute Planning & MDPs 2
3/3/2021	RL Theme: viewing party	Simons Institute Planning & MDPs 1
24/2/2021	Presentations of recent submissions	Bryn, Tim, Joost, Michael
17/2/2021	Presentations of recent submissions	Yuyang, LieTransformer, Desi, AdamF

2020

11/11 - 2/12: Theme: neural tangents and friends (Organisers: Bobby He, Yee Whye Teh)

Except for the introductory talk, remember to read the papers beforehand in order that we can discuss it in depth during the meeting.

Date	Presentation	Speakers/notes
2/12/2020	Edge of chaos in deep NNs (30min), Stable ResNets (30min)	Soufiane
25/11/2020	Deep learning vs kernel learning	Stanislav Fort
18/11/2020	Learning high frequency functions (main article), Frequency spectrum of NTKs [1] [2]	Emilien, Jef
11/11/2020	Introduction to NNGPs and neural tangent kernels (NTKs) (30 min) (NNGP1, NNGP2, NTK, linearised NNs)	Bobby He

Other papers of interest:

• https://arxiv.org/abs/2007.15801
• https://proceedings.icml.cc/static/paper_files/icml/2020/1356-Paper.pdf
• https://arxiv.org/pdf/1909.11304.pdf
• https://arxiv.org/pdf/2010.11775.pdf

Date	Presentation	Admin
11/11/2020	Analogy as Nonparametric Bayesian Inference over Relational Systems (Ruairidh) (30 min)
4/11/2020	On statistical and computational aspects of entropic optimal transport (Gonzalo Mena)
28/10/2020	Validated Variational Inference via Practical Posterior Error Bounds and more (Jun Yang)
21/10/2020	4x15min presentations of interest papers (Adam Golinski, Cong Lu, Emile Mathieu, Yee Whye Teh)
14/10/2020	30min org meeting, 2x15min presentations of interest papers (Mrinank Sharma, Michael Hutchinson)

Potential Topics

• implicit generative models.
• Gauge convolutions from Taco Cohen
• Poly-time universality and limitations of deep learning
• NN compression, quantization
• neurips exciting things

Deep generative models and related:

• Relationships between VAE / AE and [Probabilistic, Robust] PCA / Factor Analysis / ICA: nice slides, Hidden Talents of the Variational Autoencoder
• nonlinear ICA, self-supervised learning and contrastive predictive coding.
• Unbiased Implicit Variational Inference https://arxiv.org/abs/1808.02078

geometries, symmetries and relations:

• Geometric deep learning (graph convolution)
• https://deepai.org/publication/an-explicitly-relational-neural-network-architecture

GPs:

• review of GPs, scalable GPs. Kaspar / Tim

neural architectures:

• Causal inference
• meta-learning, neural processes (Jef, Jin)
• https://arxiv.org/pdf/1905.11697.pdf
• invertible models.
• Aidan: very big neural networks overview
• recent submissions (Bradley)

others:

• ABC, Likelihood-free inference e.g. this (AdamF, Bradley)
• (f/Wasserstein/Sinkhorn/MMD/Cramer)-GAN (Charline, Emile, Xenia, Antony, AdamK) and their properties: Demystifying MMD GANs, f-GAN, Wasserstein GAN, Learning Generative Models with Sinkhorn Divergences, Parametric Adversarial Divergences are Good Task Losses for Generative Modeling, Approximation and Convergence Properties of Generative Adversarial Learning
• learning CDFs and inverse CDFs (adamk)
• State-space models and TDVAE (AdamK)
• Theo Weber's big paper on credit assignment (Theo?)
• mutual information estimation: AdamF? http://bayesiandeeplearning.org/2018/papers/136.pdf
• Lagging inference network, BIVA
• Privacy preserving methods (Judith)
• Fairness & Interpretability

Guest speakers:

• Sam Smith
• Michalis Titsias

Ideas:

• Multitask learning as Multi-objective optimization

Introductory material:

Deep learning

• Supervised Learning
• Feed-Forward Neural Networks
  • Hugo Larochelle's lecture @ Deep Learning Summer School(DLSS) 2016 [slides] [video]
  • Chapter 6 of Deep Learning Book by Goodfellow, Bengio & Courville [pdf]
  • Module 1: Neural Networks of Stanford CS231n course [course website]
• Convolutional Neural Networks (CNNs)
  • Module 2: CNNs of Stanford CS231n course [course website]
  • Chapter 9 of Deep Learning Book [pdf]
  • A guide to convolution arithmetic for deep learning [pdf]
• Recurrent Neural Networks (RNNs)
  • Yoshua Bengio's lecture @ DLSS 2016 [slides] [video]
  • Chapter 10 of Deep Learning Book [pdf]
• Unsupervised Learning
• Chapter 20 of Deep Learning Book [pdf]
• Ruslan Salakhutdinov's lecture @ DLSS 2016 [slides] [video]

Probabilistic Learning

• Gatsby Probabilistic learning course [website]

Past Meetings

This reading group is a merge of the Probabilistic Inference and Deep Learning reading groups.

Hilary and Trinity 2019

Date	Presentation	Admin	Lunch Talk
16/09/2020	Break
09/09/2020	Break
02/09/2020	Cancelled
26/08/2020	Cancelled
19/08/2020	Bryn Elesedy: Improved Generalization Bound of Group Invariant / Equivariant Deep Networks via Quotient Feature Space
12/08/2020	Michael Hutchinson: Efficiently Sampling Functions from Gaussian Process Posteriors
05/08/2020	Anthony Caterini and Adam Golinkski: SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows
29/07/2020	Sheh Zaidi: Neural Ensemble Search for Performant and Calibrated Predictions
22/07/2020	Seb Farquhar: Radial Bayesian Networks
15/07/2020	ICML
08/07/2020	Bobby He: Bayesian Deep Ensembles via the Neural Tangent Kernel
01/07/2020	Maria Gorinova: Automatic Reparameterisation of Probabilistic Programs
24/06/2020	Tim Rudner: Inter-domain Deep Gaussian Processes with RKHS Fourier Features
17/06/2020	Cancelled
10/06/2020	NeurIPS Break
03/06/2020	NeurIPS Break
27/05/2020	NeurIPS Break
20/05/2020	Submissions
13/05/2020	Submissions
06/05/2020	Kaspar Martens: Initial experiences with JAX
29/04/2020	Jin Xu: Meta Learning with Implicit Gradients
22/04/2020	Adam Foster: Stein gradient estimators: A, B
15/04/2020	Adam Golinski: Learned Data Compression, Part II
08/04/2020	Joost Van Amersfoort: Compression with Flows via Local Bits-Back Coding
01/04/2020	Adam Golinski: Learned Data Compression, Part I
25/03/2020	Aidan Gomez: A Theory of Usable Information under Computational Constraints
18/03/2020	NA
11/03/2020	Bradley Gram-Hansen: Implicit Generative Models
04/03/2020	Emile Mathieu: Gauge Equivariant Convolutional Networks and the Icosahedral CNN
26/02/2020	Charline Le Lan & Emilien Dupont: Invariant & Hamiltonian Flows
19/02/2020	Michael Hutchinson: Differential Privacy
12/02/2020	Anthony Caterini: Energy inspired models with sampler induced distributions
05/02/2020	NA
29/01/2020	Bryn Elesedy: Uniform convergence may be unable to explain generalisation in deep learning
22/01/2020	Sheherya Zaidi: Robust Machine Learning
15/01/2020	Bobby He: A Primer on Natural Gradients, Tim Rudner Natural NTK, Benjie Wang: Statistically Robust Neural Network Classification	group admin

Michaelmas 2019

Date	Presentation	Admin	Lunch Talk
2/10/2019	Chemical Structure Elucidation from Mass Spectrometry by Matching Substructures Bingquan	group admin
9/10/2019	visitor: Karl Stelzner
16/10/2019	Recent submissions 1: AdamF, Yuan, Anthony+Rob
23/10/2019	visitor: Tomas Geffner
30/10/2019	Umut Simsekli
6/11/2019	Recent submissions 2: Jin, Jeff, Kaspar
13/11/2019	New students: Michael, Sheh, Bobby, Bryn
20/11/2019	Double Decent Curve (Dom Richards)
27/11/2019	Representation learning and mutual information (AdamF, AdamK, Jef)
4/12/2019	Recent submissions 3: Tim, Benjie

Trinity 2019

Date	Presenter	Topic	Material
8/5/2019	AdamF, Emile	T.S. Cohen, M. Welling, Group Equivariant Convolutional Networks. ICML 2016, T.S. Cohen, M. Geiger, J. Koehler, M. Welling, Spherical CNNs. ICLR 2018
15/5/2019	Edwin, Hyunjik	Marginal likelihood approximations, transformers
22/5/2019		no meeting (NeurIPS)
29/5/2019	Hyunjik, Kaspar	transformers continued, covariate GPLVMs
5/6/2019	Emilien, Jef, AdamG, Joost, Aidan?, Leon?	NeurIPS submissions	Rob later
12/6/2019	paused	due to ICML
19/6/2019	Xenia	VQ-VAE

Hilary 2019

Date	Presenter	Topic	Material
30/1/2019	AdamGolinski, Emile, Edwin, Tom	ICML Submissions
6/2/2019	Rob Cornish, Antony, AdamG, AdamK	ICML Submission (Rob), Flows
13/2/2019	Espen Bernton	Langevin Monte Carlo and JKO splitting
13/2/2019	Emilien	Neural ODEs, FFJORD
20/2/2019	Soufiane	On Selection of Initialisation and Activation Function of Deep NNs
20/3/2019	Jef, Jin, AdamF	Graph NNs
27/3/2019	Anthony, Ralph Abboud	Neural Tangent Kernel, Reasoning and deep learning
3/4/2019	Anthony, Bobby, Joost	Neural Tangent Kernel cont, invariance and adversarial robustness in iRevNet
24/4/2019	Emile, AdamF, Bradley/Yuan	UAI submissions and Low-level First-order PPL

MT 2018

Date	Presenter	Topic	Materials
12/10/2018	Emile	Riemannian Geometry of deep generative models	Paper
19/10/2018	Adam F	Mutual Information Estimation	Paper
26/10/2018		ICLR reviews
2/11/2018	Hyunjik	Attentive Neural Processes	Paper
9/11/2018	Jin, Charline & Emilien	Reparameterised gradients	Papers: rebar, implicit, doubly-reparameterised
16/11/2018	Hyeonwoo Noh	Transfer Learning	Paper1&Paper2
23/11/2018	Several	NeurIPS 2018 OxCSML accepted papers
30/11/2018	Several	NeurIPS 2018 accepted papers