Statistical learning theory 2022 — различия между версиями
Bbauwens (обсуждение | вклад) |
Bbauwens (обсуждение | вклад) |
||
Строка 7: | Строка 7: | ||
Seminars: Friday 18h10 -- 19h30, [https://www.hse.ru/org/persons/225526439 Artur Goldman], | Seminars: Friday 18h10 -- 19h30, [https://www.hse.ru/org/persons/225526439 Artur Goldman], | ||
− | For | + | For discussions of the materials, join the [https://t.me/+G0VKOE2-nnkwNDE0 telegram group] |
− | The course is similar to | + | The course is similar to [http://wiki.cs.hse.ru/Statistical_learning_theory_2022 last year]. |
Строка 35: | Строка 35: | ||
|| ''Part 1. Online learning'' | || ''Part 1. Online learning'' | ||
|- | |- | ||
− | | [ | + | | [2 Sept] |
− | || Lecture: philosophy. | + | || Lecture: philosophy. The online mistake bound model, the weighted majority, and perceptron algorithms [https://drive.google.com/drive/folders/1NXiLbhmO2Ml7jFmnLtjqhOgCoHg7yn9T?usp=sharing movies] |
|| [https://www.dropbox.com/s/uk9awkfa827pmtf/01allSlides.pdf?dl=0 sl01] | || [https://www.dropbox.com/s/uk9awkfa827pmtf/01allSlides.pdf?dl=0 sl01] | ||
|| [https://www.dropbox.com/s/uvsfzb997kantoa/00book_intro.pdf?dl=0 ch00] [https://www.dropbox.com/s/6ah70h5loyrz5lx/01book_onlineMistakeBound.pdf?dl=0 ch01] | || [https://www.dropbox.com/s/uvsfzb997kantoa/00book_intro.pdf?dl=0 ch00] [https://www.dropbox.com/s/6ah70h5loyrz5lx/01book_onlineMistakeBound.pdf?dl=0 ch01] | ||
− | || | + | || |
− | || | + | || |
|- | |- | ||
− | | [https://drive.google.com/file/d/16OoCqhh16BKQzyF-HM8RozigyJ3BBVxA/view?usp=sharing | + | | [https://drive.google.com/file/d/16OoCqhh16BKQzyF-HM8RozigyJ3BBVxA/view?usp=sharing 9 Sept] |
|| The perceptron algorithm in the agnostic setting. Kernels. The standard optimal algorithm. | || The perceptron algorithm in the agnostic setting. Kernels. The standard optimal algorithm. | ||
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] | || [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 sl02] | ||
|| [https://www.dropbox.com/s/0029k15cbnxj2v1/02book_sequentialOptimalAlgorithm.pdf?dl=0 ch02] [https://www.dropbox.com/s/eggk7kctgox8aza/03book_perceptron.pdf?dl=0 ch03] | || [https://www.dropbox.com/s/0029k15cbnxj2v1/02book_sequentialOptimalAlgorithm.pdf?dl=0 ch02] [https://www.dropbox.com/s/eggk7kctgox8aza/03book_perceptron.pdf?dl=0 ch03] | ||
− | || | + | || |
− | || | + | || |
|- | |- | ||
− | | | + | | 16 Sept |
|| Prediction with expert advice and the exponentially weighted majority algorithm. Recap probability theory. | || Prediction with expert advice and the exponentially weighted majority algorithm. Recap probability theory. | ||
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03] | || [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 sl03] | ||
|| [https://www.dropbox.com/s/ytl6q83q6gkax3w/04book_predictionWithExperts.pdf?dl=0 ch04] [https://www.dropbox.com/s/l11afq1d0qn6za7/05book_introProbability.pdf?dl=0 ch05] | || [https://www.dropbox.com/s/ytl6q83q6gkax3w/04book_predictionWithExperts.pdf?dl=0 ch04] [https://www.dropbox.com/s/l11afq1d0qn6za7/05book_introProbability.pdf?dl=0 ch05] | ||
− | || | + | || |
− | || | + | || |
|- | |- | ||
| | | | ||
|| ''Part 2. Distribution independent risk bounds'' | || ''Part 2. Distribution independent risk bounds'' | ||
|- | |- | ||
− | | [https://drive.google.com/file/d/1RHz8NgfianUQFlx8VswjiiPRvt0DoBvc/view?usp=sharing | + | | [https://drive.google.com/file/d/1RHz8NgfianUQFlx8VswjiiPRvt0DoBvc/view?usp=sharing 23 Sept] |
|| Sample complexity in the realizable setting, simple examples and bounds using VC-dimension | || Sample complexity in the realizable setting, simple examples and bounds using VC-dimension | ||
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04] | || [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04] | ||
|| [https://www.dropbox.com/s/8xrgcugs4xv2r2p/06book_sampleComplexity.pdf?dl=0 ch06] | || [https://www.dropbox.com/s/8xrgcugs4xv2r2p/06book_sampleComplexity.pdf?dl=0 ch06] | ||
− | || | + | || |
− | || | + | || |
|- | |- | ||
− | | [https://drive.google.com/drive/folders/1jjyJ3eIaed64ogpR11g8M44IOikt5Mj2?usp=sharing | + | | [https://drive.google.com/drive/folders/1jjyJ3eIaed64ogpR11g8M44IOikt5Mj2?usp=sharing 30 Sept] |
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | || Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | ||
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05] | || [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05] | ||
|| [https://www.dropbox.com/s/ctc48w1d2vvyiyt/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/s/jofixf9tstz0f8z/08book_VCdimension.pdf?dl=0 ch08] | || [https://www.dropbox.com/s/ctc48w1d2vvyiyt/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/s/jofixf9tstz0f8z/08book_VCdimension.pdf?dl=0 ch08] | ||
− | || | + | || |
− | || | + | || |
|- | |- | ||
− | | [https://drive.google.com/file/d/17zynIg_CZ6cCNBig5QXmBx7VFS8peyuU/view?usp=sharing | + | | [https://drive.google.com/file/d/17zynIg_CZ6cCNBig5QXmBx7VFS8peyuU/view?usp=sharing 7 Oct] |
|| Risk decomposition and the fundamental theorem of statistical learning theory | || Risk decomposition and the fundamental theorem of statistical learning theory | ||
|| [https://www.dropbox.com/s/jxijka88vfanv5n/06slides.pdf?dl=0 sl06] | || [https://www.dropbox.com/s/jxijka88vfanv5n/06slides.pdf?dl=0 sl06] | ||
|| [https://www.dropbox.com/s/r44bwxz34qj98gg/09book_riskBounds.pdf?dl=0 ch09] | || [https://www.dropbox.com/s/r44bwxz34qj98gg/09book_riskBounds.pdf?dl=0 ch09] | ||
− | || | + | || |
− | || | + | || |
|- | |- | ||
− | | | + | | 14 Oct |
|| Bounded differences inequality and Rademacher complexity | || Bounded differences inequality and Rademacher complexity | ||
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07] | || [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07] | ||
|| [https://www.dropbox.com/s/5quc1jfkrvm3t71/10book_measureConcentration.pdf?dl=0 ch10] [https://www.dropbox.com/s/km0fns8n3aihauv/11book_RademacherComplexity.pdf?dl=0 ch11] | || [https://www.dropbox.com/s/5quc1jfkrvm3t71/10book_measureConcentration.pdf?dl=0 ch10] [https://www.dropbox.com/s/km0fns8n3aihauv/11book_RademacherComplexity.pdf?dl=0 ch11] | ||
− | || | + | || |
− | || | + | || |
|- | |- | ||
| | | | ||
− | || ''Part 3. Margin risk bounds | + | || ''Part 3. Margin risk bounds with applications'' |
|- | |- | ||
− | | [https://drive.google.com/file/d/1L-BeDxhoHcoDrdlVTlfoMFwnWXKV46cr/view?usp=sharing | + | | [https://drive.google.com/file/d/1L-BeDxhoHcoDrdlVTlfoMFwnWXKV46cr/view?usp=sharing 21 Oct] |
|| Simple regression, support vector machines, margin risk bounds, and neural nets | || Simple regression, support vector machines, margin risk bounds, and neural nets | ||
|| [https://www.dropbox.com/s/0xrhe4732d0jshb/08slides.pdf?dl=0 sl08] | || [https://www.dropbox.com/s/0xrhe4732d0jshb/08slides.pdf?dl=0 sl08] | ||
|| [https://www.dropbox.com/s/cvqlwst3e69709t/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/s/dwwxgriiaj4efn0/13book_SVM.pdf?dl=0 ch13] | || [https://www.dropbox.com/s/cvqlwst3e69709t/12book_regression.pdf?dl=0 ch12] [https://www.dropbox.com/s/dwwxgriiaj4efn0/13book_SVM.pdf?dl=0 ch13] | ||
− | || | + | || |
− | || | + | || |
|- | |- | ||
− | | [https://youtu.be/9FhFxLHR4eE | + | | [https://youtu.be/9FhFxLHR4eE 4 Nov] |
|| Kernels: risk bounds, RKHS, representer theorem, design | || Kernels: risk bounds, RKHS, representer theorem, design | ||
|| [https://www.dropbox.com/s/nhqtbekclekf6k7/09slides.pdf?dl=0 sl09] | || [https://www.dropbox.com/s/nhqtbekclekf6k7/09slides.pdf?dl=0 sl09] | ||
|| [https://www.dropbox.com/s/bpb9ijn2p7k19j3/14book_kernels.pdf?dl=0 ch14] | || [https://www.dropbox.com/s/bpb9ijn2p7k19j3/14book_kernels.pdf?dl=0 ch14] | ||
− | || | + | || |
− | || | + | || |
|- | |- | ||
− | | [https://youtu.be/ZBHe5RhTuzI | + | | [https://youtu.be/ZBHe5RhTuzI 11 Nov] |
|| AdaBoost and risk bounds | || AdaBoost and risk bounds | ||
|| [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10] | || [https://www.dropbox.com/s/umum3kd9439dt42/10slides.pdf?dl=0 sl10] | ||
|| Mohri et al, chapt 7 | || Mohri et al, chapt 7 | ||
− | || | + | || |
− | || | + | || |
|- | |- | ||
| | | | ||
|| ''Part 3. Other topics'' | || ''Part 3. Other topics'' | ||
|- | |- | ||
− | | | + | | 18 Nov |
+ | || Regression I | ||
|| | || | ||
− | |||
|| | || | ||
|| | || | ||
|| | || | ||
|- | |- | ||
− | | | + | | 25 Nov |
− | || | + | || Regression II |
|| | || | ||
|| | || | ||
Строка 128: | Строка 128: | ||
|| | || | ||
|- | |- | ||
− | | | + | | 2 Dec |
− | || | + | || Multiarmed bandids I |
+ | || | ||
+ | || | ||
+ | || | ||
+ | || | ||
+ | |- | ||
+ | | 9 Dec | ||
+ | || Multiarmed bandids II | ||
|| | || | ||
|| | || | ||
Строка 135: | Строка 142: | ||
|| | || | ||
|- | |- | ||
− | | | + | | 16 Dec |
− | || Colloquium | + | || Colloquium |
|| | || | ||
|| | || |
Версия 16:56, 30 августа 2022
General Information
Lectures: Friday 16h20 -- 17h40, Bruno Bauwens, Maxim Kaledin
Seminars: Friday 18h10 -- 19h30, Artur Goldman,
For discussions of the materials, join the telegram group
The course is similar to last year.
Homeworks
Email to brbauwens-at-gmail.com. Start the subject line with SLT-HW.
Deadline before the lecture, every other lecture.
23 Sept: see problem lists 1 and 2
07 Oct: see problem lists 3 and 4
21 Oct: see problem lists 5 and 6
4 Nov: see problem list 7
18 Nov: see problem lists 8 and 9
02 Dec: see problem lists 10 and 11
Course materials
Video | Summary | Slides | Lecture notes | Problem list | Solutions |
---|---|---|---|---|---|
Part 1. Online learning | |||||
[2 Sept] | Lecture: philosophy. The online mistake bound model, the weighted majority, and perceptron algorithms movies | sl01 | ch00 ch01 | ||
9 Sept | The perceptron algorithm in the agnostic setting. Kernels. The standard optimal algorithm. | sl02 | ch02 ch03 | ||
16 Sept | Prediction with expert advice and the exponentially weighted majority algorithm. Recap probability theory. | sl03 | ch04 ch05 | ||
Part 2. Distribution independent risk bounds | |||||
23 Sept | Sample complexity in the realizable setting, simple examples and bounds using VC-dimension | sl04 | ch06 | ||
30 Sept | Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions | sl05 | ch07 ch08 | ||
7 Oct | Risk decomposition and the fundamental theorem of statistical learning theory | sl06 | ch09 | ||
14 Oct | Bounded differences inequality and Rademacher complexity | sl07 | ch10 ch11 | ||
Part 3. Margin risk bounds with applications | |||||
21 Oct | Simple regression, support vector machines, margin risk bounds, and neural nets | sl08 | ch12 ch13 | ||
4 Nov | Kernels: risk bounds, RKHS, representer theorem, design | sl09 | ch14 | ||
11 Nov | AdaBoost and risk bounds | sl10 | Mohri et al, chapt 7 | ||
Part 3. Other topics | |||||
18 Nov | Regression I | ||||
25 Nov | Regression II | ||||
2 Dec | Multiarmed bandids I | ||||
9 Dec | Multiarmed bandids II | ||||
16 Dec | Colloquium |
Problems exam
Dates, problems TBA
During the exam
-- You may consult notes, books and search on the internet
-- You may not interact with other humans (e.g. by phone, forums, etc)
The lectures in October and November are based on the book: Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. This book can be downloaded from http://gen.lib.rus.ec/ .
Office hours
Person | Monday | Tuesday | Wednesday | Thursday | Friday | |
---|---|---|---|---|---|---|
, TBA | ||||||
, TBA |
It is always good to send an email in advance. Questions and feedback are welcome.