# Extra materials for ml-mipt course

## Prerequisites

1. [en] Stanford lectures on Probability Theory:
   [link](https://web.stanford.edu/~montanar/TEACHING/Stat310A/lnotes.pdf)
1. [en] Matrix calculus notes from Stanford:
   [link](http://cs231n.stanford.edu/vecDerivs.pdf)
1. [en] Derivatives notes from Stanford:
   [link](http://cs231n.stanford.edu/handouts/derivatives.pdf)

## Basic Machine Learning

1. [en] The Hundred-page Machine Learning book: [link](http://themlbook.com)
1. [ru] Отличные лекции Жени Соколова. Читать pdf, лучше всего наиболее
   актуальный год: [link](https://github.com/esokolov/ml-course-hse)
1. [en] Naive Bayesian classifier explained:
   [link](https://machinelearningmastery.com/classification-as-conditional-probability-and-the-naive-bayes-algorithm/)
1. [en] Stanford notes on linear models:
   [link](https://see.stanford.edu/materials/aimlcs229/cs229-notes1.pdf)
1. [ru] “Рукописный учебник” от студентов нашего курса на ФИВТе:
   [link](https://github.com/ml-mipt/ml-mipt/blob/master/ML_informal_notes.pdf)
1. [ru] Методичка Воронцова,
   [link](http://www.machinelearning.ru/wiki/images/6/6d/Voron-ML-1.pdf)

## Bootstrap and bias-variance decomposition

1. [en] Detailed description of bootstrap procedure:
   [link](http://www.math.ntu.edu.tw/~hchen/teaching/LargeSample/notes/notebootstrap.pdf)
1. [en] Bias-variance tradeoff in more general case: A Unified Bias-Variance
   Decomposition and its Applications
   [link](https://homes.cs.washington.edu/~pedrod/papers/mlc00a.pdf)

## Gradient Boosting and Feature importances

1. [en] Great interactive blogpost by Alex Rogozhnikov on Gradient Boosting:
   http://arogozhnikov.github.io/2016/06/24/gradient_boosting_explained.html
1. [en] And great gradient boosted trees playground by Alex Rogozhnikov:
   http://arogozhnikov.github.io/2016/07/05/gradient_boosting_playground.html
1. [en] Shap values repo and explanation: https://github.com/slundberg/shap
1. [en] Kaggle tutorial on feature importances:
   https://www.kaggle.com/learn/machine-learning-explainability

## Deep Learning

1. [en] Deep Learning book.\
   Classical. Delivers comprehensive overview of almost all vital themes in ML and
   DL. Available online at https://www.deeplearningbook.org
1. [en] Notes on vector and matrix derivatives:
   http://cs231n.stanford.edu/vecDerivs.pdf
1. [en] More notes on matrix derivatives from Stanford:
   [link](http://cs231n.stanford.edu/handouts/derivatives.pdf)
1. [en] Stanford notes on backpropagation:
   http://cs231n.github.io/optimization-2/
1. [en] Stanford notes on different activation functions (and just intuition):
   http://cs231n.github.io/neural-networks-1/
1. [en] Great post on Medium by Andrej Karpathy:
   https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b
1. [en] CS231n notes on data preparation (batch normalization over there):
   http://cs231n.github.io/neural-networks-2/
1. [en] CS231n notes on gradient methods:
   http://cs231n.github.io/neural-networks-3/
1. [en] Original paper introducing Batch Normalization:
   https://arxiv.org/pdf/1502.03167.pdf
1. [en] What Every Computer Scientist Should Know About Floating-Point
   Arithmetic: https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
1. [en] The Unreasonable Effectiveness of Recurrent Neural Networks blog post by
   Andrej Karpathy: http://karpathy.github.io/2015/05/21/rnn-effectiveness/
1. [en] Understanding LSTM Networks:
   http://colah.github.io/posts/2015-08-Understanding-LSTMs/
1. [en] CS231n notes on data preparation:
   http://cs231n.github.io/neural-networks-2/
1. [en] Convolutional Neural Networks: Architectures, Convolution / Pooling
   Layers: http://cs231n.github.io/convolutional-networks/
1. [en] Understanding and Visualizing Convolutional Neural Networks:
   http://cs231n.github.io/understanding-cnn/
1. [en] LR warm-up and useful tricks -
   [article](https://arxiv.org/abs/1812.01187)

## Natural Language Processing

1. [en] Great resource by Lena Voita (direct link to Word Embeddings
   explanation): https://lena-voita.github.io/nlp_course/word_embeddings.html
1. [en] Word2vec tutorial:
   http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
1. [en] Beautiful post by Jay Alammar on word2vec:
   http://jalammar.github.io/illustrated-word2vec/
1. [en] Blog post about text classification with RNNs and CNNs blogpost:
   https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f
1. [en] Convolutional Neural Networks for Sentence Classification:
   https://arxiv.org/abs/1408.5882
1. [en] Great blog post by Jay Alammar on Transformer:
   https://jalammar.github.io/illustrated-transformer/
1. [en] Great Annotated Transformer article with code and comments by Harvard
   NLP group: https://nlp.seas.harvard.edu/2018/04/03/attention.html
1. [en] Harvard NLP
   [full Transformer implementation in PyTorch](http://nlp.seas.harvard.edu/2018/04/03/attention.html)
1. [en] OpenAI blog post
   [Better Language Models and Their Implications (GPT-2)](https://openai.com/blog/better-language-models/)
1. [en] Paper describing positional encoding
   ["Convolutional Sequence to Sequence Learning"](https://arxiv.org/pdf/1705.03122)
1. [en] Paper presenting [Layer Normalization](https://arxiv.org/abs/1607.06450)
1. [en] The Illustrated BERT
   [blog post](http://jalammar.github.io/illustrated-bert/)
1. [en] DistillBERT overview (distillation will be covered later in our course)
   [blog post](https://medium.com/huggingface/distilbert-8cf3380435b5)
1. [en] Google AI Blog
   [post about open sourcing BERT](https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html)
1. [en] OpenAI blog post
   [Better Language Models and Their Implications (GPT-2)](https://openai.com/blog/better-language-models/)
1. [en] One more
   [blog post explaining BERT](https://yashuseth.blog/2019/06/12/bert-explained-faqs-understand-bert-working/)
1. [en]
   [Post about GPT-2 in OpenAI blog (by 04.10.2019)](https://openai.com/blog/fine-tuning-gpt-2/)

## Graph Neural Networks

1. [en]
   [Introduction to Graph Neural Networks](https://www.morganclaypool.com/doi/10.2200/S00980ED1V01Y202001AIM045)
1. [en] Grear [repo](https://github.com/thunlp/GNNPapers) with must-read papers
   on GNN
1. [en] Reinforcement Learning: An introduction by Richard S. Sutton and Andrew
   G. Barto: [link](http://incompleteideas.net/book/the-book-2nd.html)
