Statistical learning theory 2025 — различия между версиями
Brbauwens (обсуждение | вклад) |
|||
| Строка 110: | Строка 110: | ||
|| Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. | || Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. | ||
|| | || | ||
| − | || [https://www.dropbox.com/scl/fi/ | + | || [https://www.dropbox.com/scl/fi/i30jfjbqwp3uryst3q350/16book_lossLandscapeNeuralNet.pdf?rlkey=k601mv0bcnvg79gbbbncaeipz&st=htto7z78&dl=0 ch16] |
| − | || | + | || See next |
| − | || | + | || |
|- | |- | ||
| [https://youtube.com/live/URjcCXEMPv4 02 Dec] | | [https://youtube.com/live/URjcCXEMPv4 02 Dec] | ||
| Строка 118: | Строка 118: | ||
|| | || | ||
|| [https://www.dropbox.com/scl/fi/9b3vkvxqbjbhn30mgab8z/17book_implicitRegularization.pdf?rlkey=efc6epjwi9yqr1cjb7pbhpzi3&st=l47hs8jq&dl=0 ch17] | || [https://www.dropbox.com/scl/fi/9b3vkvxqbjbhn30mgab8z/17book_implicitRegularization.pdf?rlkey=efc6epjwi9yqr1cjb7pbhpzi3&st=l47hs8jq&dl=0 ch17] | ||
| − | || | + | || [https://www.dropbox.com/scl/fi/ygnglg7z6sb40ygbvb1id/11sem.pdf?rlkey=jo7hzregeo7l0pdef3cxrqt6g&st=z5gmibhm&dl=0 prob11] |
| − | || | + | || <!-- [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11] --> |
|- | |- | ||
| 09 Dec | | 09 Dec | ||
Версия 00:33, 26 ноября 2025
Содержание
General Information
Lectures: on Tuesdays 13h00 -- 14h20 in Pokrovkaya, see here for the room a few hours before the lecture, and in zoom by Bruno Bauwens
Seminars: on Tuesdays 14h40 -- 16h00 online in Zoom by Nikita Lukianenko.
Please join the telegram group The course is similar to last year.
Homeworks
Deadline every 2 weeks, 30 min before the lecture at 12h30. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)
Before 3rd lecture, submit homework from problem lists 1 and 2. Before 5th lecture, from lists 3 and 4. Etc.
Submit homeworks in google class. You may submit preferably in English, as latex or as pictures (if in Russian, you should type it). Results are here.
Late policy: 1 homework can be submitted at most 24 hours late without explanations.
Course materials
| Video | Summary | Slides | Lecture notes | Problem list | Solutions |
|---|---|---|---|---|---|
| Part 1. Online learning | |||||
| 16 Sep | Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | sl01 | ch00 ch01 | prob01 | sol01 |
| 23 Sep | The standard optimal algorithm. The perceptron algorithm. | sl02 | ch02 ch03 | prob02 | sol02 |
| 30 Sep | Prediction with expert advice. Recap probability theory (seminar). | sl03 | ch04 ch05 | prob03 Upd 7 Oct | sol03 |
| Part 2. Distribution independent risk bounds | |||||
| 07 Oct | Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. | sl04 | ch06 | prob04 | sol04 |
| 14 Oct | Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | sl05 | ch07 ch08 | prob05 | sol05 |
| 21 Oct | Risk decomposition and the fundamental theorem of statistical learning theory (previous recording covers more) | sl06 | ch09 | prob06 | sol06 |
| 23 Oct | Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. | sl07 | ch10 ch11 | prob07 | sol07 |
| Part 3. Margin risk bounds with applications | |||||
| 06 Nov | Simple regression, support vector machines, margin risk bounds, and dropout in neural nets (switch to old recording for SVM stuff). | sl08 | ch12 ch13 | prob08 | sol08 |
| 11 Nov | Kernels: RKHS, representer theorem, risk bounds | sl09 | ch14 | prob09 | sol09 |
| 18 Nov | AdaBoost and the margin hypothesis | sl10 | ch15 | prob10 | sol10 |
| Part 4. Neural nets | |||||
| 25 Nov | Exponential (and cross entropy loss) find maximal margin solutions. Losses of neural nets are not locally convex. | ch16 | See next | ||
| 02 Dec | Lazy training and the neural tangent kernel. | ch17 | prob11 | ||
| 09 Dec | TBA | ||||
| 16 Dec | Colloquium Rules and questions previous year. Select a timeslot. |
The lectures in October and November are based on the book:
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018.
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book: Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.
Grading formula
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade.
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used.
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes.
Colloquium
Rules and questions from last year.
Date: TBA
Problems exam
Date: TBA
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc)
About questions
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity.
Office hours
Bruno Bauwens: TBA. Better send me an email in advance.
Nikita Lukianenko: Write in Telegram, the time is flexible
