The compendium provides an introduction to the theory of deep learning, from basic principles of neural network modeling and optimization to more advanced topics of neural networks as Gaussian processes, neural tangent and information theory.This unique reference text complements a largely missing theoretical introduction to neural networks without being overwhelmingly technical in a level accessible to upper-level undergraduate engineering students.Advanced chapters were designed to offer an additional intuition into the field by explaining deep learning from statistical and information theory perspectives. The book further provides additional intuition to the field by relating it to other statistical and information modeling approaches.