一种新的基于数学的神经网络分析框架，深度学

All code can be find here.

（点击尾部阅读原文前往）

This post is inspired by .

In this post, we will implement a multiple layer neural network from scratch. You can regard the number of layers and dimension of each layer as parameter. For example, `[2, 3, 2]` represents inputs with 2 dimension, one hidden layer with 3 dimension and output with 2 dimension (binary classification) (using softmax as output).

We won’t derive all the math that’s required, but I will try to give an intuitive explanation of what we are doing. I will also point to resources for you read up on the details.

Let’s start by generating a dataset we can play with. Fortunately, scikit-learn has some useful dataset generators, so we don’t need to write the code ourselves. We will go with the make_moons function.

Deep Learning第二章：线性代数

``````# Generate a dataset and plot itnp.random.seedX, y = sklearn.datasets.make_moons(200, noise=0.20)plt.scatter(X[:,0], X[:,1], s=40, c=y, cmap=plt.cm.Spectral)
``````

Deep Learning第三章：概率与信息理论

Deep Learning第四章：数值计算

Over the past decade, Deep Neural Networks have become very popular models for processing large amounts of databecause of their successful application in a wide variety of fields. Thesemodels are layered, often containing parametrized linear and non-lineartransformations at each layer in the network. At this point, however, we do notrigorously understand why DNNs are so effective. In this thesis, we explore oneway to approach this problem: we develop a generic mathematical framework forrepresenting neural networks, and demonstrate how this framework can be used torepresent specific neural network architectures. In chapter 1, we start byexploring mathematical contributions to neural networks. We can rigorouslyexplain some properties of DNNs, but these results fail to fully describe themechanics of a generic neural network. We also note that most approaches todescribing neural networks rely upon breaking down the parameters and inputsinto scalars, as opposed to referencing their underlying vector spaces, whichadds some awkwardness into their analysis. Our framework strictly operates overthese spaces, affording a more natural description of DNNs once themathematical objects that we use are well-defined and understood. We thendevelop the generic framework in chapter 3. We are able to describe analgorithm for calculating one step of gradient descent directly over the innerproduct space in which the parameters are defined. Also, we can represent theerror backpropagation step in a concise and compact form. Besides a standardsquared loss or cross-entropy loss, we also demonstrate that our framework,including gradient calculation, extends to a more complex loss functioninvolving the first derivative of the network. After developing the genericframework, we apply it to three specific network examples in chapter 4. Westart with the Multilayer Perceptron , the simplest type of DNN, and showhow to generate a gradient descent step for it. We then represent theConvolutional Neural Network , which contains more complicated inputspaces, parameter spaces, and transformations at each layer. The CNN, however,still fits into the generic framework. The last structure that we consider isthe Deep Auto-Encoder , which has parameters that are not completelyindependent at each layer. We are able to extend the generic framework tohandle this case as well. In chapter 5, we use some of the results from theprevious chapters to develop a framework for Recurrent Neural Networks ,the sequence-parsing DNN architecture. The parameters are shared across alllayers of the network, and thus we require some additional machinery todescribe RNNs. We describe a generic RNN first, and then the specific case ofthe vanilla RNN. We again compute gradients directly over inner product spaces.

The dataset we generated has two classes, plotted as red and blue points. Our goal is to train a Machine Learning classifier that predicts the correct class given the x- and y- coordinates. Note that the data is not linearly separable, we can’t draw a straight line that separates the two classes. This means that linear classifiers, such as Logistic Regression, won’t be able to fit the data unless you hand-engineer non-linear features (such as polynomials) that work well for the given dataset.

1引言与研究动机

In fact, that’s one of the major advantages of Neural Networks. You don’t need to worry about feature engineering. The hidden layer of a neural network will learn features for you.

Justin Johnson’s Python / NumPy / SciPy / Matplotlib tutorial for Stanford’s CS231n

2数学基础知识

Neural Network Architecture

You can read this tutorial () to learn the basic concepts of neural network. Like activation functions, feed-forward computation and so on.

Because we want our network to output probabilities the activation function for the output layer will be the softmax, which is simply a way to convert raw scores to probabilities. If you’re familiar with the logistic function you can think of softmax as its generalization to multiple classes.

When you choose softmax as output, you can use cross-entropy loss (also known as negative log likelihood) as loss function. More about Loss Function can be find in .

Scipy lecture notes – 涵盖了常用的各种库，介绍也比较详细，还涉及一些深入的技术话题

3神经网络的通用表达式

Learning the Parameters

Learning the parameters for our network means finding parameters (such as (W_1, b_1, W_2, b_2)) that minimize the error on our training data (loss function).

We can use gradient descent to find the minimum and I will implement the most vanilla version of gradient descent, also called batch gradient descent with a fixed learning rate. Variations such as SGD (stochastic gradient descent) or minibatch gradient descent typically perform better in practice. So if you are serious you’ll want to use one of these, and ideally you would also decay the learning rate over time.

The key of gradient descent method is how to calculate the gradient of loss function by the parameters. One approach is called Back Propagation. You can learn it more from and .

4具体的神经网络描述

Implementation

We start by given the computation graph of neural network.

In the computation graph, you can see that it contains three components (`gate`, `layer` and `output`), there is two kinds of gate (`multiply` and `add`), and you can use `tanh` layer and `softmax` output.

`gate`, `layer` and `output` can all be seen as operation unit of computation graph, so they will implement the inner derivatives of their inputs (we call it `backward`), and use chain rule according to the computation graph. You can see the following figure for nice explanation.

`gate.py`

``````import numpy as npclass MultiplyGate: def forward(self,W, X): return np.dot def backward(self, W, X, dZ): dW = np.dot(np.transpose dX = np.dot(dZ, np.transpose return dW, dXclass AddGate: def forward(self, X, b): return X   b def backward(self, X, b, dZ): dX = dZ * np.ones_like db = np.dot(np.ones((1, dZ.shape[0]), dtype=np.float64), dZ) return db, dX
``````

`layer.py`

``````import numpy as npclass Sigmoid: def forward: return 1.0 / (1.0   np.exp def backward(self, X, top_diff): output = self.forward return (1.0 - output) * output * top_diffclass Tanh: def forward: return np.tanh def backward(self, X, top_diff): output = self.forward return (1.0 - np.square * top_diff
``````

`output.py`

``````import numpy as npclass Softmax: def predict: exp_scores = np.exp return exp_scores / np.sum(exp_scores, axis=1, keepdims=True) def loss(self, X, y): num_examples = X.shape[0] probs = self.predict corect_logprobs = -np.log(probs[range(num_examples), y]) data_loss = np.sum(corect_logprobs) return 1./num_examples * data_loss def diff(self, X, y): num_examples = X.shape[0] probs = self.predict probs[range(num_examples), y] -= 1 return probs
``````

We can implement out neural network by a class `Model` and initialize the parameters in the `__init__` function. You can pass the parameter `layers_dim = [2, 3, 2]`, which represents inputs with 2 dimension, one hidden layer with 3 dimension and output with 2 dimension

``````class Model: def __init__(self, layers_dim): self.b = [] self.W = [] for i in range(len(layers_dim)-1): self.W.append(np.random.randn(layers_dim[i], layers_dim[i 1]) / np.sqrt(layers_dim[i])) self.b.append(np.random.randn(layers_dim[i 1]).reshape(1, layers_dim[i 1]))
``````

First let’s implement the loss function we defined above. It is just a forward propagation computation of out neural network. We use this to evaluate how well our model is doing:

``````def calculate_loss(self, X, y): mulGate = MultiplyGate() addGate = AddGate() layer = Tanh() softmaxOutput = Softmax() input = X for i in range(len: mul = mulGate.forward(self.W[i], input) add = addGate.forward(mul, self.b[i]) input = layer.forward return softmaxOutput.loss
``````

We also implement a helper function to calculate the output of the network. It does forward propagation as defined above and returns the class with the highest probability.

``````def predict: mulGate = MultiplyGate() addGate = AddGate() layer = Tanh() softmaxOutput = Softmax() input = X for i in range(len: mul = mulGate.forward(self.W[i], input) add = addGate.forward(mul, self.b[i]) input = layer.forward probs = softmaxOutput.predict return np.argmax(probs, axis=1)
``````

Finally, here comes the function to train our Neural Network. It implements batch gradient descent using the backpropagation algorithms we have learned above.

``````def train(self, X, y, num_passes=20000, epsilon=0.01, reg_lambda=0.01, print_loss=False): mulGate = MultiplyGate() addGate = AddGate() layer = Tanh() softmaxOutput = Softmax() for epoch in range(num_passes): # Forward propagation input = X forward = [(None, None, input)] for i in range(len: mul = mulGate.forward(self.W[i], input) add = addGate.forward(mul, self.b[i]) input = layer.forward forward.append((mul, add, input)) # Back propagation dtanh = softmaxOutput.diff(forward[len-1][2], y) for i in range(len-1, 0, -1): dadd = layer.backward(forward[i][1], dtanh) db, dmul = addGate.backward(forward[i][0], self.b[i-1], dadd) dW, dtanh = mulGate.backward(self.W[i-1], forward[i-1][2], dmul) # Add regularization terms (b1 and b2 don't have regularization terms) dW  = reg_lambda * self.W[i-1] # Gradient descent parameter update self.b[i-1]  = -epsilon * db self.W[i-1]  = -epsilon * dW if print_loss and epoch % 1000 == 0: print("Loss after iteration %i: %f" %(epoch, self.calculate_loss
``````

5递归神经网络RNN

A network with a hidden layer of size 3

Let’s see what happens if we train a network with a hidden layer size of 3.

``````import matplotlib.pyplot as pltimport numpy as npimport sklearnimport sklearn.datasetsimport sklearn.linear_modelimport mlnnfrom utils import plot_decision_boundary# Generate a dataset and plot itnp.random.seedX, y = sklearn.datasets.make_moons(200, noise=0.20)plt.scatter(X[:,0], X[:,1], s=40, c=y, cmap=plt.cm.Spectral)plt.show()layers_dim = [2, 3, 2]model = mlnn.Model(layers_dim)model.train(X, y, num_passes=20000, epsilon=0.01, reg_lambda=0.01, print_loss=True)# Plot the decision boundaryplot_decision_boundary(lambda x: model.predictplt.title("Decision Boundary for hidden layer size 3")plt.show()
``````

This looks pretty good. Our neural networks was able to find a decision boundary that successfully separates the classes.

The `plot_decision_boundary` function is referenced by .

``````import matplotlib.pyplot as pltimport numpy as np# Helper function to plot a decision boundary.def plot_decision_boundary(pred_func, X, y): # Set min and max values and give it some padding x_min, x_max = X[:, 0].min() - .5, X[:, 0].max()   .5 y_min, y_max = X[:, 1].min() - .5, X[:, 1].max()   .5 h = 0.01 # Generate a grid of points with distance h between them xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h)) # Predict the function value for the whole gid Z = pred_func(np.c_[xx.ravel(), yy.ravel Z = Z.reshape # Plot the contour and training examples plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral) plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
``````
2. We used a fixed learning rate `epsilon` for gradient descent. Implement an annealing schedule for the gradient descent learning rate (more info).
3. We used a `tanh` activation function for our hidden layer. Experiment with other activation functions (more info).
4. Extend the network from two to three classes. You will need to generate an appropriate dataset for this.
5. Try some other Parameter updates method, like `Momentum update`, `Nesterov momentum`, `Adagrad`, `RMSprop` and `Adam`(more info).
6. Some other tricks of training neural network can be find and , like `dropout reglarization`, `batch normazation`, `Gradient checks` and `Model Ensembles`.

Hugo Larochelle’s video course 这是YouTube上很火的一个深度学习视频教程，录制于2013年，但今天看内容并不过时，很详细地阐释了神经网络背后的数学理论。 幻灯片和相关资料传送门 .

6结论与未来工作展望

Stanford’s CS231n (应用于视觉识别的卷积神经网络) 由已经投奔Google的李飞飞教授和 Andrej Karpathy、Justin Johnson共同执教的课程，重点介绍了图像处理，同时也涵盖了深度学习领域的大多数重要概念。 视频 链接(2016) 、 讲义传送门

Michael Nielsen的在线著作： Neural networks and deep learning 是目前学习神经网络最容易的教材，虽然该书并未涵盖所有重要议题，但是包含大量简明易懂的阐释，同时还为一些基础概念提供了实现代码。

Ian Goodfellow、Yoshua Bengio and Aaron Courville共同编著的 Deep learning是目前深度学习领域最全面的教程资源，比其他课程涵盖的范围都要广。

Visual introduction to machine learning – decision trees

Andrew Ng’s course on machine learning, the most popular course on Coursera

Larochelle’s course doesn’t have separate introductory lectures for general machine learning, but all required concepts are defined and explained whenever needed.

1、Training and testing the models (kNN)

2、Linear classification (SVM)

4、Machine learning basics

5、Principal Component Analysis explained visually

6、How to Use t-SNE Effectively

1、Practical Machine Learning Tutorial with Python covers linear regression, k-nearest-neighbors and support vector machines. First it shows how to use them from scikit-learn, then implements the algorithms from scratch.

2、Andrew Ng’s course on Coursera has many assignments in Octave language. The same algorithms can be implemented in Python.

A Visual and Interactive Guide to the Basics of Neural Networks – shows how simple neural networks can do linear regression

1、Feedforward neural network

2、Training neural networks (up to 2.7)

3、Backpropagation

4、Architecture of neural networks

5、Using neural nets to recognize handwritten digits

6、How the backpropagation algorithm works

7、A visual proof that neural nets can compute any function

8、Deep feedforward networks

Yes you should understand backprop explains why it is important to implement backpropagation once from scratch

Calculus on computational graphs: backpropagation

Play with neural networks!

1、Implementing softmax classifier and a simple neural network in pure Python/NumPy–Jupyter notebook available

2、Andrej Karpathy implements backpropagation in Javascript in his Hacker’s guide to Neural Networks.

3、Implementing a neural network from scratch in Python

2.8-2.11. Regularization, parameter initialization etc.

7.5. Dropout

6 (first half). Setting up the data and loss

1. Improving the way neural networks learn

2. Why are deep neural networks hard to train?

3. Regularization for deep learning

4. Optimization for training deep models

5. Practical methodology

ConvNetJS Trainer demo on MNIST – visualizes the performance of different optimization algorithms

An overview of gradient descent optimization algorithms

Neural Networks, Manifolds, and Topology

Theano provides low-level primitives for constructing all kinds of neural networks. It is maintained by a machine learning group at University of Montreal. See also: Speeding up your neural network with Theano and the GPU – Jupyter notebook available

TensorFlow is another low-level framework. Its architecture is similar to Theano. It is maintained by the Google Brain team.

Torch is a popular framework that uses Lua language. The main disadvantage is that Lua’s community is not as large as Python’s. Torch is mostly maintained by Facebook and Twitter.

There are also higher-level frameworks that run on top of these:

Lasagne is a higher level framework built on top of Theano. It provides simple functions to create large networks with few lines of code.

Keras is a higher level framework that works on top of either Theano or TensorFlow.

1. Computer vision (up to 9.9)

6 (second half). Intro to ConvNets

1. Convolutional neural networks

2. Localization and detection

3. Visualization, Deep dream, Neural style, Adversarial examples

4. Image segmentation (up to 38:00) includes upconvolutions

5. Deep learning

6. Convolutional networks

Image Kernels explained visually – shows how convolutional filters (also known as image kernels) transform the image

ConvNetJS MNIST demo – live visualization of a convolutional network right in the browser

Conv Nets: A Modular Perspective

Understanding Convolutions

Understanding Convolutional neural networks for NLP

Theano: Convolutional Neural Networks (LeNet)

Using Lasagne for training Deep Neural Networks

Detecting diabetic retinopathy in eye images – a blog post by one of the best performers of Diabetic retinopathy detection contest in Kaggle. Includes a good example of data augmentation.

Face recognition for right whales using deep learning – the authors used different ConvNets for localization and classification. Code and models are available.

Tensorflow: Convolutional neural networks for image classification on CIFAR-10 dataset

Implementing a CNN for text classification in Tensorflow

DeepDream implementation in TensorFlow

92.45% on CIFAR-10 in Torch – implements famous VGGNet network with batch normalization layers in Torch

Training and investigating Residual Nets – Residual networks perform very well on image classification tasks. Two researchers from Facebook and CornellTech implemented these networks in Torch

ConvNets in practice – lots of practical tips on using convolutional networks including data augmentation, transfer learning, fast implementations of convolution operation

The Unreasonable Effectiveness of Recurrent Neural Networks – describes how RNNs can generate text, math papers and C code

Hugo Larochelle’s course doesn’t cover recurrent neural networks (although it covers many topics that RNNs are used for). We suggest watching Recurrent Neural Nets and LSTMs by Nando de Freitas to fill the gap

1. Recurrent Neural Networks, Image Captioning, LSTM

2. Soft attention (starting at 38:00)

Michael Nielsen’s book stops at convolutional networks. In the Other approaches to deep neural nets section there is just a brief review of simple recurrent networks and LSTMs.

1. Sequence Modeling: Recurrent and Recursive Nets

Recurrent neural networks from Stanford’s CS224d (2016) by Richard Socher

Understanding LSTM Networks

Theano: Recurrent Neural Networks with Word Embeddings

Theano: LSTM Networks for Sentiment Analysis

Implementing a RNN with Python, Numpy and Theano

Lasagne implementation of Karpathy’s char-rnn

Combining CNN and RNN for spoken language identification in Lasagne

Automatic transliteration with LSTM using Lasagne

Tensorflow: Recurrent Neural Networks for language modeling

Recurrent Neural Networks in Tensorflow

Understanding and Implementing Deepmind’s DRAW Model

LSTM implementation explained

Torch implementation of Karpathy’s char-rnn

Autoencoders

Autoencoder是为非监督式学习设计的神经网络，例如当数据没有标记的情况。Autoencoder可以用来进行数据维度消减，以及为其他神经网络进行预训练，以及数据生成等。以下课程资源中，我们还收录了Autoencoder与概率图模型整合的一个autoencoders的变种，其背后的数学机理在下一章“概率图模型”中会介绍。

1. Autoencoder

7.6. Deep autoencoder

1. Videos and unsupervised learning (from 32:29) – this video also touches an exciting topic of generative adversarial networks.

2. Autoencoders

ConvNetJS Denoising Autoencoder demo

Karol Gregor on Variational Autoencoders and Image Generation

Autoencoder的部署

Theano: Denoising autoencoders

Diving Into TensorFlow With Stacked Autoencoders

Variational Autoencoder in TensorFlow

Training Autoencoders on ImageNet Using Torch 7

Building autoencoders in Keras

1. Conditional Random Fields

2. Training CRFs

3. Restricted Boltzman machine

7.7-7.9. Deep Belief Networks

9.10. Convolutional RBM

1. Linear Factor Models – first steps towards probabilistic models

2. Structured Probabilistic Models for Deep Learning

3. Monte Carlo Methods

4. Confronting the Partition Function

5. Approximate Inference

6. Deep Generative Models – includes Boltzmann machines (RBM, DBN, …), variational autoencoders, generative adversarial networks, autoregressive models etc.

Generative models – a blog post on variational autoencoders, generative adversarial networks and their improvements by OpenAI.

The Neural Network Zoo attempts to organize lots of architectures using a single scheme.

Restricted Boltzmann Machines in Theano

Deep Belief Networks in Theano

Generating Large Images from Latent Vectors – uses a combination of variational autoencoders and generative adversarial networks.

Image Completion with Deep Learning in TensorFlow – another application of generative adversarial networks.

Generating Faces with Torch – Torch implementation of Generative Adversarial Networks

Arxiv Sanity Preserver 为浏览 arXiv上的论文提供了一个漂亮的界面.

Videolectures.net 含有大量关于深度学习的高级议题视频

/r/MachineLearning 一个非常活跃的Reddit分支. 几乎所有重要的新论文这里都有讨论。