Statistical learning theory 2024/25

General Information

Lectures: on Tuesday 9h30--10h50 in room M302 and in zoom by Bruno Bauwens

Seminars: online in Zoom by Nikita Lukianenko.

Please join the telegram group The course is similar to last year.

Homeworks

Deadline every 2 weeks, before the lecture. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)

Before 3rd lecture, submit HW from problem lists 1 and 2. (Thus Oct 1st, 9h30.) Before 5th lecture, from lists 3 and 4. Etc.

Classroom to submit homeworks. Results are here.

Late policy: 1 homework can be submitted at most 24 late without explanations.

Course materials

Video	Summary	Slides	Lecture notes	Problem list	Solutions
	Part 1. Online learning
21 Sep	Philosophy. The online mistake bound model. The halving and weighted majority algorithms.	sl01	ch00 ch01	prob01	sol01
24 Sep	The standard optimal algorithm. The perceptron algorithm.	sl02	ch02 ch03	prob02	sol02
01 Oct	Kernel perceptron algorithm. Prediction with expert advice. Recap probability theory (seminar).	sl03	ch04 ch05	prob03	sol03
	Part 2. Distribution independent risk bounds
08 Oct	Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes.	sl04	ch06	prob05
15 Oct	Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions	sl05	ch07 ch08	prob06
22 Oct	Risk decomposition and the fundamental theorem of statistical learning theory	sl06	ch09	prob07
29 Oct	Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma.	sl07	ch10 ch11	prob08
	Part 3. Margin risk bounds with applications
05 Nov	Simple regression, support vector machines, margin risk bounds, and neural nets with dropout regularization	sl08	ch12 ch13	prob09
12 Nov	Kernels: RKHS, representer theorem, risk bounds	sl09	ch14	prob10
19 Nov	AdaBoost and the margin hypothesis	sl10	ch15	prob11
26 Nov	Implicit regularization of stochastic gradient descent in overparameterized neural nets (recording with many details about the Hessian)		ch16 ch17
03 Dec	Part 2 of previous lecture: Hessian control and stability of the NTK.

The lectures in October and November are based on the book: Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018.

A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book: Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.

Grading formula

Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.

All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade.

There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used.

Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes.

Colloquium

Rules and questions from last year.

Date: TBA

Problems exam

TBA
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc)

Office hours

Bruno Bauwens: Bruno Bauwens: Tuesday 12h -- 20h. Wednesday 16h -- 18h. Friday 11h -- 17h. Better send me an email in advance.

Nikita Lukianenko: Write in Telegram, the time is flexible