Statistical learning theory 2025 — различия между версиями
Brbauwens (обсуждение | вклад) |
Brbauwens (обсуждение | вклад) |
||
| Строка 31: | Строка 31: | ||
|| Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | || Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | ||
|| [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01] | || [https://www.dropbox.com/scl/fi/954v3zn1d3zn68crzr85l/01slides_all.pdf?rlkey=7b613hrqwbho2qqtoekj8ua3s&st=lcg46pnb&dl=0 sl01] | ||
| − | || [https://www.dropbox.com/scl/fi/ | + | || [https://www.dropbox.com/scl/fi/x07bx3n4col196mm3twnv/00book_intro.pdf?rlkey=zexicpaviliqm8141n056h61z&st=puhs63f2&dl=0 ch00] [https://www.dropbox.com/scl/fi/uqa9615215wy7ievgr50y/01book_onlineMistakeBound.pdf?rlkey=jiqzz84b5ipaw4t6cff7b17sl&st=mc354l04&dl=0 ch01] |
|| [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01] | || [https://www.dropbox.com/scl/fi/37lvjuq06v3yaejqsbn4v/01sem.pdf?rlkey=7940pxuyduvrinz0639axglx7&st=jt3lchhd&dl=0 prob01] | ||
|| [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01] | || [https://www.dropbox.com/scl/fi/kswtqmyxw3pv336g1vdd6/01sol.pdf?rlkey=bpwnrcsj6ru3nbo4xwq2lp6g0&st=hftnu87m&dl=0 sol01] | ||
Версия 18:53, 16 сентября 2025
Содержание
General Information
Lectures: on Tuesdays 11h10 -- 12h30 in room S222 and in zoom by Bruno Bauwens
Seminars: on Tuesdays 13h00 -- 14h20 online in Zoom by Nikita Lukianenko.
Please join the telegram group The course is similar to last year.
Homeworks
Deadline every 2 weeks, before the lecture. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)
Before 3rd lecture, submit homework from problem lists 1 and 2. Before 5th lecture, from lists 3 and 4. Etc.
Use --this link-- to submit homeworks. You may submit in English or Russian, as latex or as pictures. Results are here.
Late policy: 1 homework can be submitted at most 24 late without explanations.
Course materials
| Video | Summary | Slides | Lecture notes | Problem list | Solutions |
|---|---|---|---|---|---|
| Part 1. Online learning | |||||
| 16 Sep | Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | sl01 | ch00 ch01 | prob01 | sol01 |
| ?? Sep | The standard optimal algorithm. The perceptron algorithm. | sl02 | ch02 ch03 | prob02 | |
| ?? Oct | Kernel perceptron algorithm. Prediction with expert advice. Recap probability theory (seminar). | sl03 | ch04 ch05 | prob03 | |
| Part 2. Distribution independent risk bounds | |||||
| ?? Oct | Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. | sl04 | ch06 | prob04 | |
| ?? Oct | Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | sl05 | ch07 ch08 | prob05 | |
| ?? Oct | Risk decomposition and the fundamental theorem of statistical learning theory (previous recording covers more) | sl06 | ch09 | prob06 | |
| ?? Nov | Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. | sl07 | ch10 ch11 | prob07 | |
| Part 3. Margin risk bounds with applications | |||||
| ?? Nov | Simple regression, support vector machines, margin risk bounds, and neural nets with dropout regularization | sl08 | ch12 ch13 | prob08 | |
| ?? Nov | Kernels: RKHS, representer theorem, risk bounds | sl09 | ch14 | prob09 | |
| ?? Nov | AdaBoost and the margin hypothesis | sl10 | ch15 | prob10 | |
| ?? Dec | Losses of neural nets are not locally convex. Gradient descent with stable gradients. (Old recording about Hessians) | ch16 | prob11 | ||
| ?? Dec | Lazy training and the neural tangent kernel. | ch17 |
The lectures in October and November are based on the book:
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018.
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book: Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.
Grading formula
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade.
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used.
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes.
Colloquium
Rules and questions from last year.
Date: TBA
Problems exam
Date: TBA
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc)
About questions
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity.
Office hours
Bruno Bauwens: TBA. Better send me an email in advance.
Nikita Lukianenko: Write in Telegram, the time is flexible
