Statistical learning theory 2020 — различия между версиями

Материал из Wiki - Факультет компьютерных наук
Перейти к: навигация, поиск
(Новая страница: « == General Information == Lectures: Saturday 9h30 - 10h50, zoom https://zoom.us/j/96210489901 Teachers: Bruno Bauwens and Vladimir Podolskii Seminar for group…»)
 
 
(не показаны 124 промежуточные версии этого же участника)
Строка 2: Строка 2:
 
== General Information ==
 
== General Information ==
  
−
Lectures: Saturday 9h30 - 10h50, zoom https://zoom.us/j/96210489901
+
[https://www.dropbox.com/s/a8dd33ousa76qad/grading.pdf?dl=0 Grading]
  
 
Teachers: Bruno Bauwens and Vladimir Podolskii
 
Teachers: Bruno Bauwens and Vladimir Podolskii
  
−
Seminar for group 1: Saturday 11h10 - 12h30, Bruno Bauwens and Vladimir Podolskii zoom https://zoom.us/j/94186131884,
+
Lectures: Saturday 9h30 - 10h50, zoom https://zoom.us/j/96210489901
  
−
Seminar for group 2: Tuesday ??, Nikita Lukyanenko
+
Seminars <br>
 +
- Group 1: Saturday 11h10 - 12h30, Bruno Bauwens and Vladimir Podolskii zoom https://zoom.us/j/94186131884, <br>
 +
- Group 2: Tuesday 18h, Nikita Lukyanenko, see [https://ruz.hse.ru ruz.hse.ru]
 +
 
 +
 
 +
== Reexam ==
 +
 
 +
Date: Sat 23 Jan and 30 Jan 14h
 +
 
 +
Consists of a retake of the colloquium and the problems exam. Somewhere in the first hour (depending on the availability of the teacher), you redo the colloquium and in the remaining time, you solve 4 or 5 problems similar as in the exam on Dec 23rd.
 +
 
 +
In the calculation of the grades, the homework results are dropped, and the final grade consists of the average of the colloquium part and the problems part (with equal weight).
 +
 
 +
Zoom link 30 Jan: <span style="color:red">[https://zoom.us/j/99399187196]</span>
 +
 
 +
 
 +
== Colloquium ==
 +
 
 +
Saturday 12 Dec and Tuesday 15 Dec, online. Choose your [https://docs.google.com/spreadsheets/d/1tuBV6H_NdwRiR1YJdmv2vZLY0aFpM-Xh9DqZtYdTxm8/edit#gid=0 timeslot]
 +
 
 +
[https://www.dropbox.com/s/0f4v5vp0fsz7e34/colloqQuest.pdf?dl=0 Rules and questions.] version 06/12. [https://www.dropbox.com/s/ugiqfsk2mg01262/QandA.pdf?dl=0 Q&A]
  
 
== Course materials ==
 
== Course materials ==
Строка 14: Строка 34:
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
−
! Date !! Summary !! Lecture notes !! Problem list !! Solutions
+
! Date !! Summary !! Lecture notes !! Slides !! Video !! Problem list !! Solutions
 
|-
 
|-
−
| 12 Sept || Introduction and sample complexity in the realizable setting || [https://www.dropbox.com/s/l8e8xjfe2f8tjz8/01lect.pdf?dl=0 lecture1.pdf]  
+
| 12 Sept || Introduction and sample complexity in the realizable setting  
−
|| [https://www.dropbox.com/s/kicoo9xf356eam5/01lect.pdf?dl=0 Problem list 1] || [https://www.dropbox.com/s/cixli4sghy0w01q/01solution.pdf?dl=0 Solutions 1]
+
|| [https://www.dropbox.com/s/kicoo9xf356eam5/01lect.pdf?dl=0 lecture1.pdf]  
 +
|| [https://www.dropbox.com/s/pehka8xyu5hlpis/slides01.pdf?dl=0 slides1.pdf]
 +
||
 +
|| [https://www.dropbox.com/s/fbdew1vdzskenie/01sem.pdf?dl=0 Problem list 1] <span style="color:red">Update 26.09, prob 1.7</span>
 +
|| [https://www.dropbox.com/s/rn8nv9y0db61a0h/01sol.pdf?dl=0 Solutions 1]
 
|-
 
|-
−
| 19 Sept || VC-dimension and sample complexity ||  
+
| 19 Sept || VC-dimension and sample complexity  
 +
|| [https://www.dropbox.com/s/ayry6kp91h5s1nv/02lect.pdf?dl=0 lecture2.pdf]
 +
|| [https://www.dropbox.com/s/6p6h1ooy4i5wt1t/02slides.pdf?dl=0 slides2.pdf]
 +
|| [https://youtu.be/SBoffzKZebg Chapt 2,3]
 +
|| [https://www.dropbox.com/s/4qn4qzr6mgu9lt3/02sem.pdf?dl=0 Problem list 2]
 +
|| [https://www.dropbox.com/s/0g5gw3yrjjjzz07/02sol.pdf?dl=0 Solutions 2]
 
|-
 
|-
−
| 26 Sept || Risk bounds and the fundamental theorem of statistical learning theory ||  
+
| 26 Sept || Risk bounds and the fundamental theorem of statistical learning theory  
 +
|| [https://www.dropbox.com/s/njekia6g8t0x5mb/03lect.pdf?dl=0 lecture3.pdf]
 +
|| [https://www.dropbox.com/s/at4eph4mv9gfnp1/03slides.pdf?dl=0 slides3.pdf]
 +
||
 +
|| [https://www.dropbox.com/s/nvb25e0ccebbz2a/03sem.pdf?dl=0 Problem list 3]
 +
|| [https://www.dropbox.com/s/5jbl0xul25mrbg1/03sol.pdf?dl=0 Solutions 3]
 
|-
 
|-
−
| 03 Nov || Rademacher complexity and margin assumption ||
+
| 03 Oct || Rademacher complexity  
−
<!---
+
|| [https://www.dropbox.com/s/ggw79gau85a4mcl/04lect.pdf?dl=0 lecture4.pdf]
 +
|| [https://www.dropbox.com/s/pd2ockzxqdfo66t/04slides.pdf?dl=0 slides4.pdf]
 +
||
 +
|| [https://www.dropbox.com/s/rbx6jwlusnwhkzn/04sem.pdf?dl=0 Problem list 4] <span style="color:red">Update 23.10, prob 4.1d</span>
 +
|| [https://www.dropbox.com/s/nhxkxfjajzsgfnf/04sol.pdf?dl=0 Solutions 4]
 
|-
 
|-
−
| 1 Oct || Agnostic PAC-learnability is equivalent with finite VC-dimension, structural risk minimization || [https://www.dropbox.com/s/jsrse5qaqk2jhi1/05lect.pdf?dl=0 lecture5.pdf] 14/10 || [https://www.dropbox.com/s/etw67uq1pu5g58t/05sem.pdf?dl=0 Problem list 5] || [https://www.dropbox.com/s/6mpom53yrldcrjy/05solution.pdf?dl=0 Solution 5]
+
| 10 Oct || Support vector machines and risk bounds
 +
|| Chapt 5, Mohri et al, see below
 +
|| [https://www.dropbox.com/s/q2onm9o6wgceg5i/05slides.pdf?dl=0 slides5.pdf]
 +
||
 +
|| [https://www.dropbox.com/s/upv70of97fqpx5f/05sem.pdf?dl=0 Problem list 5] <span style="color:red">Update 29.10, typo 5.8</span>
 +
|| [https://www.dropbox.com/s/jfneptto1qoug1g/05sol.pdf?dl=0 Solutions 5]
 
|-
 
|-
−
| 9 Oct || Boosting, Mohri's book pages 121-131. || [https://www.dropbox.com/s/m6tc4miryv6cs21/06lect.pdf?dl=0 lecture6.pdf] 23/10 || [https://www.dropbox.com/s/85t74k9wmibcnmr/06sem.pdf?dl=0 Problem list 6] || No solution.
+
| 17 Oct || Support vector machines and recap
 +
|| Chapt 5, Mohri et al.
 +
|| [https://www.dropbox.com/s/tot9akaoonja1zp/06slides.pdf?dl=0 slides6.pdf]
 +
||
 +
|| [https://www.dropbox.com/s/y7w3srgsrp9d7m0/06sem.pdf?dl=0 Problem list 6] <span style="color:red">Update 10.11</span>
 +
|| [https://www.dropbox.com/s/qc0847q8q8llgg2/06sol.pdf?dl=0 Solutions 6] 
 
|-
 
|-
−
| 15 Oct || Rademacher complexity and contraction lemma (=Talagrand's lemma), Mohri's book pages 33-41 and 78-79 || [https://www.dropbox.com/s/y2vr3mrwp66cuvz/07lect.pdf?dl=0 lecture7.pdf] || [https://www.dropbox.com/s/cuo0tmfv4k2egvh/07sem.pdf?dl=0 Problem list 7] || See lecture7.pdf
+
| 31 Oct || Kernels
 +
|| [https://www.dropbox.com/s/lzhbe7sb4aw49d4/07lec.pdf?dl=0 lecture7.pdf]
 +
|| [https://www.dropbox.com/s/yrptkeaydam7r2v/07slides.pdf?dl=0 slides7.pdf]
 +
||
 +
|| [https://www.dropbox.com/s/81edvzrgiel3do6/07sem.pdf?dl=0 Problem list 7] <span style="color:red">Update 11.11, prob 7.6</span>
 +
|| [https://www.dropbox.com/s/xaoxh2i12x15jz6/07sol.pdf?dl=0 Solutions 7]
 
|-
 
|-
−
| 21 Oct || Margin theory and risk bounds for boosting. || [https://www.dropbox.com/s/o5zae3d8nw5eexw/08lect.pdf?dl=0 lecture8.pdf] || [https://www.dropbox.com/s/xg7u3ss1a0vog5j/08sem.pdf?dl=0 Problem list 8]|| See lecture6.pdf for ex. 8.6.
+
| 07 Nov || Adaboost
 +
|| Chapt 6, Mohri et al
 +
|| [https://www.dropbox.com/s/2ied3qr0xrsb127/08slides.pdf?dl=0 slides8.pdf]
 +
||
 +
|| [https://www.dropbox.com/s/i9jo9dlj06t51um/08sem.pdf?dl=0 Problem list 8]
 +
|| [https://www.dropbox.com/s/1bxxzvorzbxpgji/08sol.pdf?dl=0 Solutions 8]
 
|-
 
|-
−
| 12 Nov || Deep boosting, we study the paper [http://www.cs.nyu.edu/~mohri/pub/mboost.pdf Multi-class deep boosting], V. Kuznetsov, M Mohri, and U. Syed, Advances in Neural Information Processing Systems, p2501--2509, 2014. Notes will be provided. || [https://www.dropbox.com/s/tc7drmxwu53opzq/09lect.pdf?dl=0 lecture9.pdf] || [https://www.dropbox.com/s/lsu6tgmc767u3yd/09sem.pdf?dl=0 Problem list 9] || [https://www.dropbox.com/s/8wmswbynzx0s9hd/09sol.pdf?dl=0 Solutions 9.]
+
| 14 Nov || Online learning 1, Littlestone dimension, weighted majority algorithm
 +
|| Chapt 7, Mohri et al, and [http://machinelearning.ru/wiki/images/9/99/SLT%2C_lecture_85.pdf Животовский]
 +
|| [https://www.dropbox.com/s/rtlsy6ssm2yj2p0/09slides.pdf?dl=0 slides9.pdf]
 +
||
 +
|| [https://www.dropbox.com/s/k0ynyl5x874e0gq/09sem.pdf?dl=0 Problem list 9] <span style="color:red">Update 08.12, 9.4</span>
 +
|| [https://www.dropbox.com/s/k2zpqnoiwe19osu/09sol.pdf?dl=0 Solutions 9]
 
|-
 
|-
−
| 19 Nov || Support vector machines, primal and dual optimization problem, risk bounds.  || See chapt. 5 of Mohri's book || [https://www.dropbox.com/s/ys37nsdfz3aa4ry/10sem.pdf?dl=0 Problem list 10]|| No solution.
+
| 21 Nov || Online learning 2, Exponential weighted average algorithm, preceptron
 +
|| Chapt 7, Mohri et al
 +
|| [https://www.dropbox.com/s/rtlsy6ssm2yj2p0/09slides.pdf?dl=0 slides9.pdf]
 +
||
 +
|| [https://www.dropbox.com/s/jh7krrihpc5f3ua/10sem.pdf?dl=0 Problem list 10]
 +
|| [https://www.dropbox.com/s/tf8mdjxfbz86lj4/10sol.pdf?dl=0 Solutions 10]
 
|-
 
|-
−
| 26 Nov || Kernels, Kernel reproducing Hilbert spaces, representer theorem, examples of kernels || [https://www.dropbox.com/s/xkic1j6r516ierl/11lect.pdf?dl=0 lecture11.pdf] || [https://www.dropbox.com/s/g3huq5aqzdaesrg/11sem.pdf?dl=0 Problem set 11] || Solutions: see lecture11.pdf
+
| 28 Nov || Online learning 3, perception, Winnow and online to batch conversion
 +
|| Chapt 7, Mohri et al
 +
|| [https://www.dropbox.com/s/ntkmnxhsvk9j38y/11slides.pdf?dl=0 slides11.pdf]
 +
||
 +
|| [https://www.dropbox.com/s/py43d5k4mr7rv26/11sem.pdf?dl=0 Problem list 11]
 +
|| [https://www.dropbox.com/s/fuj1wclaq7wwa7c/11sol.pdf?dl=0 Solutions 11]
 
|-
 
|-
−
| 3 Dec || A polynomial time improper learning algorithm for constant depth L1-regularized neural networks, from [http://www.jmlr.org/proceedings/papers/v48/zhangd16.pdf this paper].  Online algorithms: halving algorithm, weighted and exponentially weighted average algorithms. See Mohri's book Sections 7.1 and 7.2. || [https://www.dropbox.com/s/aq6798jps111l86/12lect.pdf?dl=0 lecture12.pdf] || [https://www.dropbox.com/s/o4t6smc70o1bt3t/12sem.pdf?dl=0 Problem list 12] || No solution.
+
| 5 Dec || Recap of requested topics, Q&A
 +
|| [https://www.dropbox.com/s/ugiqfsk2mg01262/QandA.pdf?dl=0 Q&A]
 +
||  
 +
||
 +
||
 +
||
 
|-
 
|-
−
| 10 Dec || We finish online learning. Discuss the algorithm from [http://papers.nips.cc/paper/4616-bandit-algorithms-boost-brain-computer-interfaces-for-motor-task-selection-of-a-brain-controlled-button.pdf this paper].  || || See previous list. ||
 
 
|}
 
|}
−
-->
 
  
−
<-- A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:
+
 
 +
<!-- A gentle introduction to the materials of the first 3 lectures and an overview of probability theory, can be found in chapters 1-6 and 11-12 of the following book:
 
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.-->
 
Sanjeev Kulkarni and Gilbert Harman: An Elementary Introduction to Statistical Learning Theory, 2012.-->
  
−
<-- Afterward, we hope to cover chapters 1-8 from the book:
+
The lectures in October and November are based on the book:
−
Foundations of machine learning, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2012. These books can be downloaded from http://gen.lib.rus.ec/ .
+
Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. This book can be downloaded from http://gen.lib.rus.ec/ .
  
−
(We will study a new boosting algorithm, based on the paper: )
+
For online learning, we also study a few topics from  [http://machinelearning.ru/wiki/index.php?title=%D0%A2%D0%B5%D0%BE%D1%80%D0%B8%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B8%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8F_(%D0%BA%D1%83%D1%80%D1%81_%D0%BB%D0%B5%D0%BA%D1%86%D0%B8%D0%B9%2C_%D0%9D._%D0%9A._%D0%96%D0%B8%D0%B2%D0%BE%D1%82%D0%BE%D0%B2%D1%81%D0%BA%D0%B8%D0%B9) lecture notes] by Н. К. Животовский
−
-->
+
  
 
== Office hours ==
 
== Office hours ==
Строка 61: Строка 137:
 
! Person !! Monday !! Tuesday !! Wednesday !! Thursday !! Friday !!  
 
! Person !! Monday !! Tuesday !! Wednesday !! Thursday !! Friday !!  
 
|-
 
|-
−
|  Bruno Bauwens ||   || || || ||  || Room&nbsp;620
+
|  [https://www.hse.ru/en/org/persons/160550073 Bruno Bauwens], [https://zoom.us/j/5579743402 Zoom] (email in advance) || 14h-18h  || 16h15-20h || || ||  || Room&nbsp;S834 Pokrovkaya 11
 
|-
 
|-
 
|}
 
|}
  
 +
It is always good to send an email in advance. Questions are welcome.
  
 +
<!--
 
== Russian texts  ==
 
== Russian texts  ==
  
Строка 72: Строка 150:
 
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru]  
 
``Наука и искусство построения алгоритмов, которые извлекают знания из данных'', Петер Флах. On [http://www.machinelearning.ru machinelearning.ru]  
 
you can find brief and clear definitions.
 
you can find brief and clear definitions.
 +
-->

Текущая версия на 10:02, 3 сентября 2021

General Information

Grading

Teachers: Bruno Bauwens and Vladimir Podolskii

Lectures: Saturday 9h30 - 10h50, zoom https://zoom.us/j/96210489901

Seminars
- Group 1: Saturday 11h10 - 12h30, Bruno Bauwens and Vladimir Podolskii zoom https://zoom.us/j/94186131884,
- Group 2: Tuesday 18h, Nikita Lukyanenko, see ruz.hse.ru


Reexam

Date: Sat 23 Jan and 30 Jan 14h

Consists of a retake of the colloquium and the problems exam. Somewhere in the first hour (depending on the availability of the teacher), you redo the colloquium and in the remaining time, you solve 4 or 5 problems similar as in the exam on Dec 23rd.

In the calculation of the grades, the homework results are dropped, and the final grade consists of the average of the colloquium part and the problems part (with equal weight).

Zoom link 30 Jan: [1]


Colloquium

Saturday 12 Dec and Tuesday 15 Dec, online. Choose your timeslot

Rules and questions. version 06/12. Q&A

Course materials

Date Summary Lecture notes Slides Video Problem list Solutions
12 Sept Introduction and sample complexity in the realizable setting lecture1.pdf slides1.pdf Problem list 1 Update 26.09, prob 1.7 Solutions 1
19 Sept VC-dimension and sample complexity lecture2.pdf slides2.pdf Chapt 2,3 Problem list 2 Solutions 2
26 Sept Risk bounds and the fundamental theorem of statistical learning theory lecture3.pdf slides3.pdf Problem list 3 Solutions 3
03 Oct Rademacher complexity lecture4.pdf slides4.pdf Problem list 4 Update 23.10, prob 4.1d Solutions 4
10 Oct Support vector machines and risk bounds Chapt 5, Mohri et al, see below slides5.pdf Problem list 5 Update 29.10, typo 5.8 Solutions 5
17 Oct Support vector machines and recap Chapt 5, Mohri et al. slides6.pdf Problem list 6 Update 10.11 Solutions 6
31 Oct Kernels lecture7.pdf slides7.pdf Problem list 7 Update 11.11, prob 7.6 Solutions 7
07 Nov Adaboost Chapt 6, Mohri et al slides8.pdf Problem list 8 Solutions 8
14 Nov Online learning 1, Littlestone dimension, weighted majority algorithm Chapt 7, Mohri et al, and Животовский slides9.pdf Problem list 9 Update 08.12, 9.4 Solutions 9
21 Nov Online learning 2, Exponential weighted average algorithm, preceptron Chapt 7, Mohri et al slides9.pdf Problem list 10 Solutions 10
28 Nov Online learning 3, perception, Winnow and online to batch conversion Chapt 7, Mohri et al slides11.pdf Problem list 11 Solutions 11
5 Dec Recap of requested topics, Q&A Q&A


The lectures in October and November are based on the book: Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. This book can be downloaded from http://gen.lib.rus.ec/ .

For online learning, we also study a few topics from lecture notes by Н. К. Животовский

Office hours

Person Monday Tuesday Wednesday Thursday Friday
Bruno Bauwens, Zoom (email in advance) 14h-18h 16h15-20h Room S834 Pokrovkaya 11

It is always good to send an email in advance. Questions are welcome.


Источник — «http://wiki.cs.hse.ru/index.php?title=Statistical_learning_theory_2020&oldid=57173»