6  Deep Neural Networks

If a two-layer network can already approximate everything, why go deeper? to be deep is to be flexible.

in high dimensional spaces, most of the probability mass is located in a thin shell at a specific radius

forward vs inverse problem. A single NN can deal with one but not the other (see red fitted curve)

6.1 Limitations of fixed basis functions

6.1.1 The curse of dimensionality

6.1.2 High dimensional spaces

shells

6.1.3 Data manifolds

digits

6.1.4 Data-dependent basis functions

kernel methods

6.2 Multilayer networks

6.2.1 Parameter matrices

6.2.2 Universal approximation

with examples

6.2.3 Hidden unit activation functions

softmax

6.2.4 weight-space symmetries

6.3 Deep networks

6.3.1 Hierarchical representations

6.3.2 Distributed representations

6.3.3 Representation learning

6.3.4 Transfer learning

6.3.5 Contrastive learning

6.3.6 General network architectures

6.3.7 Tensors

6.4 Error functions

6.4.1 Regression

6.4.2 Binary classification

6.4.3 Multiclass classification

6.5 Mixture density networks

6.5.1 Robot kinematics example

6.5.2 Conditional mixture distributions

6.5.3 Gradient optimization

6.5.4 Predictive distribution