Statistical learning theory 2023/24 — различия между версиями

Материал из Wiki - Факультет компьютерных наук
Перейти к: навигация, поиск
Строка 119: Строка 119:
 
|-  
 
|-  
 
| [https://youtu.be/GL574ljefJ8 05 Dec]
 
| [https://youtu.be/GL574ljefJ8 05 Dec]
|| Implicit regularization of stochastic gradient descent in neural nets
+
|| Implicit regularization of stochastic gradient descent in overparameterized neural nets
 
||  
 
||  
 
|| [https://www.dropbox.com/s/b4xac5uki7l1ysq/16book_implicitRegularization.pdf?dl=0 ch16]
 
|| [https://www.dropbox.com/s/b4xac5uki7l1ysq/16book_implicitRegularization.pdf?dl=0 ch16]
Строка 129: Строка 129:
 
|-
 
|-
 
| 09 Dec
 
| 09 Dec
|| Optional. Saturday 14h40, seminar 16h20. Double descent and neural nets at initialization.  
+
|| Optional. Saturday 14h40, seminar 16h20. Double descent and wide neural nets at initialization.  
 
||  
 
||  
 
||  
 
||  
Строка 139: Строка 139:
 
|-
 
|-
 
| 16 Dec
 
| 16 Dec
|| Optional. Saturday 14h40, seminar 16h20. Gradient flows and evolution of neural tangent kernels.  
+
|| Optional. Saturday 14h40, seminar 16h20. Evolution of NTK during stochastic gradient descent.  
 
||
 
||
 
||
 
||

Версия 13:10, 20 сентября 2023

General Information

Lectures: Tuesday 14h40 -- 16h00, room S321 and in zoom by Bruno Bauwens and Maxim Kaledin,

Seminars: Monday 16h20 -- 17h40, room N506, and in zoom by Artur Goldman.

To discuss the materials, join the telegram group The course is similar to last year.


Homeworks

Deadline every 2 weeks, before the seminar at 16h00. Homework problems from

seminars 1 and 2 on September 25, seminars 3 and 4 on October 9, seminars 5 and 6 on October 23, seminars 7 and 8 on November 13, seminars 9 and 10 on November 27, seminars 11 and 12 before the start of the exam.

Email to brbauwens-at-gmail.com. Start the subject line with SLT-HW. -Link with results todo-.


Course materials

Video Summary Slides Lecture notes Problem list Solutions
Part 1. Online learning
05 Sept Philosophy. The online mistake bound model. The halving and weighted majority algorithms movies sl01 ch00 ch01 prob01 sol01
12 Sept The perceptron algorithm. Kernels. The standard optimal algorithm. sl02 ch02 ch03 prob02 sol02
19 Sept Prediction with expert advice. Recap probability theory (seminar). sl03 ch04 ch05
26 Sept Multi-armed bandids.
03 Oct Multi-armed bandids.
Part 2. Distribution independent risk bounds
10 Oct Sample complexity in the realizable setting, simple examples and bounds using VC-dimension sl04 ch06
17 Oct Growth functions, VC-dimension and the characterization of sample comlexity with VC-dimensions sl05 ch07 ch08
24 Oct Risk decomposition and the fundamental theorem of statistical learning theory sl06 ch09
07 Nov Bounded differences inequality, Rademacher complexity, symmetrization, contraction lemma. sl07 ch10 ch11
Part 3. Margin risk bounds with applications
14 Nov Simple regression, support vector machines, margin risk bounds, and neural nets sl08 ch12 ch13
21 Nov Kernels: RKHS, representer theorem, risk bounds sl09 ch14
28 Nov AdaBoost and the margin hypothesis sl10 ch15
05 Dec Implicit regularization of stochastic gradient descent in overparameterized neural nets ch16
Part 4. Neural tangent kernels
09 Dec Optional. Saturday 14h40, seminar 16h20. Double descent and wide neural nets at initialization.
12 Dec Colloquium (you may choose between 12 Dec and 19 Dec).
16 Dec Optional. Saturday 14h40, seminar 16h20. Evolution of NTK during stochastic gradient descent.
19 Dec Colloquium

Background on multi-armed bandits: A. Slivkins, [Introduction to multi-armed bandits https://arxiv.org/pdf/1904.07272.pdf], 2022.

The lectures in October and November are based on the book: Foundations of machine learning 2nd ed, Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalker, 2018. This book can be downloaded from Library Genesis (the link changes sometimes and sometimes vpn is needed).


Grading formula

Final grade = 0.35 * [score of homeworks] + 0.35 * [score of colloquium] + 0.3 * [score on the exam] + bonus from quizzes.

All homework questions have the same weight. Each solved extra homework task increases the score of the final exam by 1 point.

There is no rounding except on the final grade. Grades fractional grades above 5/10 are rounded up, those below 5/10 are rounded down.

Autogrades: if you only need 4/10 to pass with maximal final score, it will be given automatically. This may happen because of extra questions and bonuses from quizzes.


Colloquium

Rules and questions of previous year.


Problems exam

December 21--30, TBA.
-- You may use handwritten notes, lecture materials from this wiki (either printed or through your PC), Mohri's book
-- You may not search on the internet or interact with other humans (e.g. by phone, forums, etc)


Office hours

Bruno Bauwens: Wednesday 13h-16h, Friday 14h-20h, (better send an email in advance).

Maxim Kaledin: Write in Telegram, the time is flexible