Statistical learning theory 2024/25 — различия между версиями
Bauwens (обсуждение | вклад) |
Bauwens (обсуждение | вклад) |
||
| (не показано 25 промежуточных версии этого же участника) | |||
| Строка 6: | Строка 6: | ||
Please join the [https://t.me/+1begXb8SomhmODI8 telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2023/24 last year]. | Please join the [https://t.me/+1begXb8SomhmODI8 telegram group] The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2023/24 last year]. | ||
| + | |||
| + | [https://docs.google.com/spreadsheets/d/1eo4OvNObJicoY-sfMTMUg7dhjbgcpfkGDs9dIehpUA0/edit?usp=sharing Results] | ||
| + | |||
| + | == Problems exam == | ||
| + | |||
| + | Friday 20 December 11h-14h, D507<br> | ||
| + | -- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book <br> | ||
| + | -- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc) | ||
| + | |||
| + | About questions<br> | ||
| + | -- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)<br> | ||
| + | -- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity. | ||
== Homeworks == | == Homeworks == | ||
| Строка 87: | Строка 99: | ||
|| [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13] | || [https://www.dropbox.com/s/573a2vtjfx8qqo8/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/scl/fi/hxeh5btc0bb2f52fnqh5f/13book_SVM.pdf?rlkey=dw3u2rtfstpsb8mi9hnuc8poy&dl=0 ch13] | ||
|| [https://www.dropbox.com/scl/fi/ekrdaba2gzpxdp58yvfwo/08sem.pdf?rlkey=vsljva82ekk6ol6k7w1g87pz6&st=146i9y67&dl=0 prob08] | || [https://www.dropbox.com/scl/fi/ekrdaba2gzpxdp58yvfwo/08sem.pdf?rlkey=vsljva82ekk6ol6k7w1g87pz6&st=146i9y67&dl=0 prob08] | ||
| − | || | + | || [https://www.dropbox.com/scl/fi/fcu1kbczqnxjbvtjpxst7/08sol.pdf?rlkey=irlhu14q6d12poymmc25xmh6q&st=pt7euz9i&dl=0 sol08] |
|- | |- | ||
| − | | [https:// | + | | [https://youtube.com/live/77-rZFzX2O8 19 Nov] |
|| Kernels: RKHS, representer theorem, risk bounds | || Kernels: RKHS, representer theorem, risk bounds | ||
|| [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09] | || [https://www.dropbox.com/s/jst60ww8ev4ypie/09slides.pdf?dl=0 sl09] | ||
|| [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14] | || [https://www.dropbox.com/scl/fi/lozpqk5nnm8us77qfhn7x/14book_kernels.pdf?rlkey=s8e7a46rm3znkw13ubj3fzzz0&dl=0 ch14] | ||
| − | || [https://www.dropbox.com/scl/fi/ | + | || [https://www.dropbox.com/scl/fi/t7jv4gulwbdluc278sadi/09sem.pdf?rlkey=wzitr8cwastoq5koyvpsj252o&st=cdik5cp7&dl=0 prob09] |
| − | || | + | || [https://www.dropbox.com/scl/fi/2pxx6ctc7qv4xpvc4esla/09sol.pdf?rlkey=dg9pncbr6d294gz5me3efzrwp&st=v49ksm24&dl=0 sol09] |
|- | |- | ||
| [https://www.youtube.com/watch?v=OgiaWrWh_WA 26 Nov] | | [https://www.youtube.com/watch?v=OgiaWrWh_WA 26 Nov] | ||
|| AdaBoost and the margin hypothesis | || AdaBoost and the margin hypothesis | ||
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10] | || [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10] | ||
| − | || [https://www.dropbox.com/ | + | || [https://www.dropbox.com/scl/fi/ef1ti9gagjv49mdky1364/15book_AdaBoost.pdf?rlkey=h6myd1zxm74quktq1cy2rc2ae&st=r2at7eha&dl=0 ch15] |
| − | || [https://www.dropbox.com/scl/fi/ | + | || [https://www.dropbox.com/scl/fi/y3mbpbcoau67i1nfjg7lr/10sem.pdf?rlkey=mfye4kcfgm9gf6aos6z8nd6q4&st=n1btlv8c&dl=0 prob10] |
| − | || | + | || [https://www.dropbox.com/scl/fi/5lbthnkjkn35y68ohmhm4/10sol.pdf?rlkey=0w0twp97ohfrlcsspnzfg0wgh&st=74hhghgd&dl=0 sol10] |
|- | |- | ||
| − | | [https:// | + | | [https://youtube.com/live/DUgksR6gOQ8 03 Dec] |
| − | || | + | || Losses of neural nets are not locally convex. Gradient descent with stable gradients. ([https://www.youtube.com/watch?v=ygVHVW3y3wM Old recording] about Hessians) |
|| | || | ||
| − | || [https://www.dropbox.com/scl/fi/ | + | || [https://www.dropbox.com/scl/fi/ipsngdfvo4bvhofxh4377/16book_lossLandscapeNeuralNet.pdf?rlkey=3018bx9wczc4rpu7xq0wxdc2q&st=64mz3r2p&dl=0 ch16] |
| + | || [https://www.dropbox.com/scl/fi/dc86iowe91nlzf3fu1h71/11sem.pdf?rlkey=87a7uqqpy4n39bcm3dxbsidew&st=t1gemioe&dl=0 prob11] | ||
| + | || [https://www.dropbox.com/scl/fi/topptsvelhdpog2qucfpr/11sol.pdf?rlkey=ceev18140kz2ly8y8crxixf03&st=lvk4j2rz&dl=0 sol11] | ||
| + | |- | ||
| + | | [https://youtube.com/live/URjcCXEMPv4 10 Dec] | ||
| + | || Lazy training and the neural tangent kernel. | ||
|| | || | ||
| + | || [https://www.dropbox.com/scl/fi/9b3vkvxqbjbhn30mgab8z/17book_implicitRegularization.pdf?rlkey=efc6epjwi9yqr1cjb7pbhpzi3&st=l47hs8jq&dl=0 ch17] | ||
|| | || | ||
| + | || | ||
|- | |- | ||
| − | | [https://www. | + | | 17 Dec |
| − | + | || Colloquium 9h30 - 12h30 (room D725) and 18h10 - 21h (different building Старая Басманная А-125). [https://www.dropbox.com/scl/fi/e2692ns95pg0kj0m4e0wo/colloqQuest.pdf?rlkey=peey4u0dxz0vohv39a3oc67ft&st=c87t9kqu&dl=0 Rules and questions.] Reserve in [https://docs.google.com/spreadsheets/d/17pJaioWm3Vo2aYB2J3msTyxj9NSovbZHOC_w6Cxvsj8/edit?usp=sharing shedule.] | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
|} | |} | ||
| Строка 141: | Строка 156: | ||
Date: TBA | Date: TBA | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
== Office hours == | == Office hours == | ||
| − | Bruno Bauwens: Bruno Bauwens: Tuesday 12h -- 20h | + | Bruno Bauwens: Bruno Bauwens: Tuesday 12h -- 20h. Friday 15h -- 17h30. Better send me an email in advance. |
Nikita Lukianenko: Write in Telegram, the time is flexible | Nikita Lukianenko: Write in Telegram, the time is flexible | ||
Текущая версия на 16:41, 18 декабря 2024
Содержание
[убрать]General Information
Lectures: on Tuesday 9h30--10h50 in room M302 and in zoom by Bruno Bauwens
Seminars: online in Zoom by Nikita Lukianenko.
Please join the telegram group The course is similar to last year.
Problems exam
Friday 20 December 11h-14h, D507
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc)
About questions
-- 4 questions of the difficulty of the homework. (Many homework questions were from former exams.)
-- I always ask to calculate VC dimension and to give/prove some risk bound with Rademacher complexity.
Homeworks
Deadline every 2 weeks, before the lecture. The tasks are at the end of each problem list. (Problem lists will be updated, check the year.)
Before 3rd lecture, submit HW from problem lists 1 and 2. Before 5th lecture, from lists 3 and 4. Etc.
Classroom to submit homeworks. You may submit in English or Russian, as latex or as pictures. Results are here.
Late policy: 1 homework can be submitted at most 24 late without explanations.
Course materials
| Video | Summary | Slides | Lecture notes | Problem list | Solutions |
|---|---|---|---|---|---|
| Part 1. Online learning | |||||
| 21 Sep | Philosophy. The online mistake bound model. The halving and weighted majority algorithms. | sl01 | ch00 ch01 | prob01 | sol01 |
| 24 Sep | The standard optimal algorithm. The perceptron algorithm. | sl02 | ch02 ch03 | prob02 | sol02 |
| 01 Oct | Kernel perceptron algorithm. Prediction with expert advice. Recap probability theory (seminar). | sl03 | ch04 ch05 | prob03 | sol03 |
| Part 2. Distribution independent risk bounds | |||||
| 08 Oct | Necessity of a hypothesis class. Sample complexity in the realizable setting, examples: threshold functions and finite classes. | sl04 | ch06 | prob04 update 12.10 | sol04 |
| 15 Oct | Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | sl05 | ch07 ch08 | prob05 | |
| 22 Oct | Risk decomposition and the fundamental theorem of statistical learning theory (previous recording covers more) | sl06 | ch09 | prob06 | sol06 |
| 05 Nov | Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. | sl07 | ch10 ch11 | prob07 | sol07 |
| Part 3. Margin risk bounds with applications | |||||
| 12 Nov | Simple regression, support vector machines, margin risk bounds, and neural nets with dropout regularization | sl08 | ch12 ch13 | prob08 | sol08 |
| 19 Nov | Kernels: RKHS, representer theorem, risk bounds | sl09 | ch14 | prob09 | sol09 |
| 26 Nov | AdaBoost and the margin hypothesis | sl10 | ch15 | prob10 | sol10 |
| 03 Dec | Losses of neural nets are not locally convex. Gradient descent with stable gradients. (Old recording about Hessians) | ch16 | prob11 | sol11 | |
| 10 Dec | Lazy training and the neural tangent kernel. | ch17 | |||
| 17 Dec | Colloquium 9h30 - 12h30 (room D725) and 18h10 - 21h (different building Старая Басманная А-125). Rules and questions. Reserve in shedule. |
The lectures in October and November are based on the book:
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018.
A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book: Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.
Grading formula
Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.
All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point. At the end of the lectures there is a short quiz in which you may earn 0.1 bonus points on the final non-rounded grade.
There is no rounding except for transforming the final grade to the official grade. Arithmetic rounding is used.
Autogrades: if you only need 6/10 on the exam to have the maximal 10/10 for the course, this will be given automatically. This may happen because of extra homework questions and bonuses from quizzes.
Colloquium
Rules and questions from last year.
Date: TBA
Office hours
Bruno Bauwens: Bruno Bauwens: Tuesday 12h -- 20h. Friday 15h -- 17h30. Better send me an email in advance.
Nikita Lukianenko: Write in Telegram, the time is flexible
