Statistical learning theory 2021 — различия между версиями

Материал из Wiki - Факультет компьютерных наук
Перейти к: навигация, поиск
(не показаны 44 промежуточные версии 2 участников)
Строка 6: Строка 6:
 
Teachers: [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens] and [https://www.hse.ru/en/org/persons/225553845 Nikita Lukianenko]  
 
Teachers: [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens] and [https://www.hse.ru/en/org/persons/225553845 Nikita Lukianenko]  
  
Lectures: Saturday 14:40 - 16:00. The lectures are Pokrovkaya and also streamed [https://us02web.zoom.us/j/82173400975?pwd=L1lhTzFTc2lGem5BVFdRcFEyVUhqZz09  here] in zoom.
+
Lectures: Saturday 14:40 - 16:00. The lectures are Pokrovkaya and also streamed [https://us02web.zoom.us/j/82173400975?pwd=L1lhTzFTc2lGem5BVFdRcFEyVUhqZz09  here] in zoom.  
  
Seminars: Tuesday 16:20 - 17:40. The seminars are Pokrovkaya and also streamed [https://zoom.us/j/95201626157?pwd=dE16MHlJU0xtakNUTHdvRE5POEdpQT09 here] in zoom.
+
Seminars: Tuesday 16:20 - 17:40. The seminars are Pokrovkaya and also streamed [https://meet.google.com/ber-yzns-hxz here] in google.meet.
  
 
See [https://ruz.hse.ru/ruz/main ruz] for the rooms.
 
See [https://ruz.hse.ru/ruz/main ruz] for the rooms.
  
Practical information on [https://t.me/joinchat/IER2-8hc0wUxNDQ0 telegram group]
+
Practical information on a telegram group. [https://t.me/joinchat/IER2-8hc0wUxNDQ0 Join here.]
  
 
The course is similar [http://wiki.cs.hse.ru/Statistical_learning_theory_2020 last year], except for the order of topics and part 3.
 
The course is similar [http://wiki.cs.hse.ru/Statistical_learning_theory_2020 last year], except for the order of topics and part 3.
Строка 18: Строка 18:
 
== Homeworks ==
 
== Homeworks ==
  
Email to brbauwens-at-gmail.com.  
+
Email to brbauwens-at-gmail.com. Start the subject line with SLT-HW. [https://www.dropbox.com/s/taskzhu0nj5motd/scores.ods?dl=0 Results]
  
Deadline before the lecture, every 2 weeks.
+
Deadline before the lecture, every other lecture.
  
<span style="color:red">25 Sept</span>: see problem lists 1 and 2 <span style="color:red">[update 9 Sept]</span>
+
25 Sept: see problem lists 1 and 2  
  
09 Oct: see problem lists 3 and 4
+
09 Oct: see problem lists 3 <span style="color:red">Update 30/9</span> and 4
  
 
Etc.
 
Etc.
Строка 35: Строка 35:
 
|-
 
|-
 
|  
 
|  
|| ''Part 1. Online learning'' || || ||
+
|| ''Part 1. Online learning''  
 
|-
 
|-
 
| [https://drive.google.com/file/d/1WL9LSNDD1B_q6LdpfDQ8BPluNfhjWrD9/view?usp=sharing 4 Sept]  
 
| [https://drive.google.com/file/d/1WL9LSNDD1B_q6LdpfDQ8BPluNfhjWrD9/view?usp=sharing 4 Sept]  
Строка 48: Строка 48:
 
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 02sl]  
 
|| [https://www.dropbox.com/s/sy959ee81mov5cr/02slides.pdf?dl=0 02sl]  
 
|| [https://www.dropbox.com/s/0029k15cbnxj2v1/02book_sequentialOptimalAlgorithm.pdf?dl=0 02ch] [https://www.dropbox.com/s/eggk7kctgox8aza/03book_perceptron.pdf?dl=0 03ch]
 
|| [https://www.dropbox.com/s/0029k15cbnxj2v1/02book_sequentialOptimalAlgorithm.pdf?dl=0 02ch] [https://www.dropbox.com/s/eggk7kctgox8aza/03book_perceptron.pdf?dl=0 03ch]
|| [https://www.dropbox.com/s/415nws7qi589bme/02sem.pdf?dl=0 02prob]
+
|| [https://www.dropbox.com/s/415nws7qi589bme/02sem.pdf?dl=0 02prob (23 Sept)]
||
+
|| [https://www.dropbox.com/s/ofcctflbnxt0kx3/02sol.pdf?dl=0 02sol]
 
|-
 
|-
| 18 Sept
+
| 18 Sept (rec to do)
|| Prediction with expert advice and the exponentially weighted majority algorithm. Recap probability theory. Leave on out risk for SVM.
+
|| Prediction with expert advice and the exponentially weighted majority algorithm. Recap probability theory.  
||
+
|| [https://www.dropbox.com/s/a60p9b76cxusgqy/03slides.pdf?dl=0 03sl]
||
+
|| [https://www.dropbox.com/s/ytl6q83q6gkax3w/04book_predictionWithExperts.pdf?dl=0 ch04] [https://www.dropbox.com/s/l11afq1d0qn6za7/05book_introProbability.pdf?dl=0 ch05]
||
+
|| [https://www.dropbox.com/s/nsrcy3yxgey67lp/03sem.pdf?dl=0 03prob(30 Sept)]
||
+
|| [https://www.dropbox.com/s/bg9nd01h1fhzjsi/03sol.pdf?dl=0 03sol]
 
|-
 
|-
 
|  
 
|  
|| ''Part 2. Supervised classification'' || || ||
+
|| ''Part 2. Risk bounds for binary classification''  
 
|-
 
|-
| 25 Sept
+
| [https://drive.google.com/file/d/1RHz8NgfianUQFlx8VswjiiPRvt0DoBvc/view?usp=sharing 25 Sept]
|| Sample complexity in the realizable setting, simple example and bounds using VC-dimension
+
|| Sample complexity in the realizable setting, simple examples and bounds using VC-dimension
||
+
|| [https://www.dropbox.com/s/pi0f3wab1xna6d7/04slides.pdf?dl=0 sl04]
||
+
|| [https://www.dropbox.com/s/8xrgcugs4xv2r2p/06book_sampleComplexity.pdf?dl=0 ch06]
||
+
|| [https://www.dropbox.com/s/7qn2yz5fxc93rez/04sem.pdf?dl=0 04prob]
||
+
|| [https://www.dropbox.com/s/xm3nhgj5d6h49nz/04sol.pdf?dl=0 04sol]
 
|-  
 
|-  
 
| 2 Oct
 
| 2 Oct
 +
|| Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions
 +
|| [https://www.dropbox.com/s/rpnh6288rdb3j8m/05slides.pdf?dl=0 sl05]
 +
|| [https://www.dropbox.com/s/ctc48w1d2vvyiyt/07book_growthFunctions.pdf?dl=0 ch07] [https://www.dropbox.com/s/jofixf9tstz0f8z/08book_VCdimension.pdf?dl=0 ch08]
 +
|| [https://www.dropbox.com/s/zbyqxy3qp3pz79i/05sem.pdf?dl=0 05prob]
 +
|| [https://www.dropbox.com/s/a8efm18dof2zeox/05sol.pdf?dl=0 05sol]
 +
|-
 +
| [https://drive.google.com/file/d/17zynIg_CZ6cCNBig5QXmBx7VFS8peyuU/view?usp=sharing 9 Oct]
 
|| Risk decomposition and the fundamental theorem of statistical learning theory
 
|| Risk decomposition and the fundamental theorem of statistical learning theory
||
+
|| [https://www.dropbox.com/s/jxijka88vfanv5n/06slides.pdf?dl=0 sl06]
||
+
|| [https://www.dropbox.com/s/r44bwxz34qj98gg/09book_riskBounds.pdf?dl=0 ch09]
||
+
|| [https://www.dropbox.com/s/x87txc8v5v6u8vb/06sem.pdf?dl=0 06prob]
||
+
|| [https://www.dropbox.com/s/ydlqu8oce3xj6ix/06sol.pdf?dl=0 06sol]
 
|-
 
|-
| 9 Oct
+
| 16 Oct
|| Rademacher complexity
+
|| Bounded differences inequality and Rademacher complexity
||
+
|| [https://www.dropbox.com/s/kfithyq0dgcq6h8/07slides.pdf?dl=0 sl07]
||
+
|| [https://www.dropbox.com/s/5quc1jfkrvm3t71/10book_measureConcentration.pdf?dl=0 ch10] [https://www.dropbox.com/s/km0fns8n3aihauv/11book_RademacherComplexity.pdf?dl=0 ch11]
 
||
 
||
 
||
 
||
 
|-
 
|-
| 16 Oct
+
| 29 Oct
 
|| Support vector machines and margin risk bounds
 
|| Support vector machines and margin risk bounds
 
||  
 
||  
Строка 89: Строка 96:
 
||
 
||
 
|-
 
|-
| 29 Oct
+
| 6 Nov
 
|| Kernels: risk bounds, design, and representer theorem
 
|| Kernels: risk bounds, design, and representer theorem
 
||
 
||
Строка 96: Строка 103:
 
||
 
||
 
|-  
 
|-  
| 6 Nov
+
| 13 Nov
 
|| AdaBoost and risk bounds
 
|| AdaBoost and risk bounds
 
||
 
||
Строка 104: Строка 111:
 
|-
 
|-
 
|
 
|
|| ''Part 3. Other topics'' || || ||
+
|| ''Part 3. Other topics''  
 
|-
 
|-
| 13 Nov  
+
| 20 Nov  
 
|| Clustering   
 
|| Clustering   
 
||  
 
||  
Строка 113: Строка 120:
 
||  
 
||  
 
|-
 
|-
| 20 Nov
+
| 27 Nov
 
|| Dimensionality reduction and the Johnson-Lindenstrauss lemma
 
|| Dimensionality reduction and the Johnson-Lindenstrauss lemma
 
||
 
||
Строка 120: Строка 127:
 
||
 
||
 
|-
 
|-
| 27 Nov
+
| 4 Dec
 
|| Active learning
 
|| Active learning
||
 
||
 
||
 
||
 
|-
 
| 4 Dec
 
|| Extra space for a lesson, in the likely case we are a bit slower.
 
 
||
 
||
 
||
 
||
Строка 242: Строка 242:
 
|-
 
|-
 
|  [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens], [https://zoom.us/j/5579743402 Zoom] ||  || 12h30-14h30 || || || 14h-20h || Room&nbsp;S834 Pokrovkaya 11
 
|  [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens], [https://zoom.us/j/5579743402 Zoom] ||  || 12h30-14h30 || || || 14h-20h || Room&nbsp;S834 Pokrovkaya 11
 +
|-
 +
|  [https://www.hse.ru/org/persons/225553845 Nikita Lukianenko], [https://t.me/vaulty Telegram] ||  || 14h30-16h30 || 14h30-16h30 || || || Room&nbsp;S831 Pokrovkaya 11
 
|-
 
|-
 
|}
 
|}
  
It is always good to send an email in advance. Questions and feedback are welcome.
+
It is always good to send an email in advance. Questions and feedback are welcome.
 +
 
 +
I am traveling from Sept 12 -- Sept 30 and Oct 16 -- Oct 26. On Fridays I'm available till 16h30.  
  
 
<!--
 
<!--

Версия 01:13, 25 октября 2021

General Information

Grading

Teachers: Bruno Bauwens and Nikita Lukianenko

Lectures: Saturday 14:40 - 16:00. The lectures are Pokrovkaya and also streamed here in zoom.

Seminars: Tuesday 16:20 - 17:40. The seminars are Pokrovkaya and also streamed here in google.meet.

See ruz for the rooms.

Practical information on a telegram group. Join here.

The course is similar last year, except for the order of topics and part 3.

Homeworks

Email to brbauwens-at-gmail.com. Start the subject line with SLT-HW. Results

Deadline before the lecture, every other lecture.

25 Sept: see problem lists 1 and 2

09 Oct: see problem lists 3 Update 30/9 and 4

Etc.

Course materials

Video Summary Slides Lecture notes Problem list Solutions
Part 1. Online learning
4 Sept Lecture: philosophy. Seminar: the online mistake bound model, the weighted majority, and perceptron algorithms movies 01sl 00ch 01ch 01prob (9 Sept) 01sol
11 Sept The perceptron algorithm in the agnostic setting. Kernels. The standard optimal algorithm. 02sl 02ch 03ch 02prob (23 Sept) 02sol
18 Sept (rec to do) Prediction with expert advice and the exponentially weighted majority algorithm. Recap probability theory. 03sl ch04 ch05 03prob(30 Sept) 03sol
Part 2. Risk bounds for binary classification
25 Sept Sample complexity in the realizable setting, simple examples and bounds using VC-dimension sl04 ch06 04prob 04sol
2 Oct Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions sl05 ch07 ch08 05prob 05sol
9 Oct Risk decomposition and the fundamental theorem of statistical learning theory sl06 ch09 06prob 06sol
16 Oct Bounded differences inequality and Rademacher complexity sl07 ch10 ch11
29 Oct Support vector machines and margin risk bounds
6 Nov Kernels: risk bounds, design, and representer theorem
13 Nov AdaBoost and risk bounds
Part 3. Other topics
20 Nov Clustering
27 Nov Dimensionality reduction and the Johnson-Lindenstrauss lemma
4 Dec Active learning
11 Dec Colloquium


The lectures in October and November are based on the book: Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. This book can be downloaded from http://gen.lib.rus.ec/ .


Office hours

Person Monday Tuesday Wednesday Thursday Friday
Bruno Bauwens, Zoom 12h30-14h30 14h-20h Room S834 Pokrovkaya 11
Nikita Lukianenko, Telegram 14h30-16h30 14h30-16h30 Room S831 Pokrovkaya 11

It is always good to send an email in advance. Questions and feedback are welcome.

I am traveling from Sept 12 -- Sept 30 and Oct 16 -- Oct 26. On Fridays I'm available till 16h30.