笔者使用如何阅读深度学习论文的方法进行阅读,文中标注的 $1(第一步)、$2、$3、$4 分别表示在第该步阅读中的记录和思考

注:为了加深理解,大家可以根据使用 TensorFlow 2 Keras 实现 AlexNet CNN 架构实践 AlexNet 网络。

ImageNet Classification with Deep Convolutional Neural Networks

使用深度卷积神经网络的 ImageNet 分类

$1 本论文的重点是使用深度卷积神经网络ImageNet 分类,可以猜到该网络应该就是 AlexNet



We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.

我们训练了一个大型的深度卷积神经网络ImageNet LSVRC-2010 竞赛中的 120 万个的高分辨率图像分成 1000 个不同的类别。在测试数据上,我们实现了 top-1 37.5%top-5 17.0% 的错误率,这比以往的最新技术要好得多。这个神经网络具有 6000 万个参数和 650,000 个神经元,由 5卷积层(某些卷积层后面带有最大池化层)和 3全连接层以及最后一个 1000softmax 组成。为了使训练更快,我们使用了非饱和神经元并对卷积操作进行了非常有效的 GPU 实现。为了减少全连接中的过拟合,我们采用了最近开发的被称为 “dropout” 的正则化方法,事实证明这是非常有效的。我们也使用这个模型的一个变种参加了 ILSVRC-2012 竞赛,赢得了冠军,取得了 top-5 15.3% 的错误率,相比之下,第二名为 top-5 26.2%

$1 摘要中说明了 AlexNet 是个大型的网络,包含卷积层、最大池化层、全连接层、softmax。并且,作者使用了非饱和神经元、卷积操作 GPU 实现、dropout 等操作来优化网络,在几次大赛中都获得不错的成绩。

1 Introduction

1 简介

Current approaches to object recognition make essential use of machine learning methods. To im-prove their performance, we can collect larger datasets, learn more powerful models, and use bet-ter techniques for preventing overfitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the current-best error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to col-lect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories.

当前的目标识别方法基本上都使用了机器学习方法。为了提高目标识别的性能,我们可以收集更大的数据集,学习更强大的模型,使用更好的技术来防止过拟合。直到最近,标注图像的数据集都相对较小–在几万张图像的数量级上(例如,NORB[16],Caltech-101/256 [8, 9]和 CIFAR-10/100 [12])。简单的识别任务在这样大小的数据集上可以被解决的相当好,尤其是如果通过标签保留变换进行数据增强的情况下。例如,目前在 MNIST 数字识别任务上(<0.3%)的最好准确率已经接近了人类水平[4]。但真实环境中的对象表现出了相当大的可变性,因此为了学习识别它们,有必要使用更大的训练数据集。实际上,小图像数据集的缺点已经被广泛认识到(例如,Pinto et al. [21]),但收集上百万图像的标注数据仅在最近才变得的可能。新的更大的数据集包括 LabelMe [23],它包含了数十万张完全分割的图像,ImageNet[6],它包含了 22000 个类别上的超过 1500 万张标注的高分辨率的图像。

To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this prob-lem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be con-trolled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.

为了从数百万张图像中学习几千个对象,我们需要一个有很强学习能力的模型。然而对象识别任务的巨大复杂性意味着这个问题不能被指定,即使通过像 ImageNet 这样的大数据集,因此我们的模型应该也有许多先验知识来补偿我们所没有的数据。卷积神经网络(CNNs)构成了一个这样的模型[16, 11, 13, 18, 15, 22, 26]。它们的能力可以通过改变它们的广度深度来控制,它们也可以对图像的本质进行强大且通常正确的假设(也就是说,统计的稳定性和像素依赖的局部性)。因此,与具有层次大小相似的标准前馈神经网络,CNNs 有更少的连接和参数,因此它们更容易训练,而它们理论上的最佳性能可能仅比标准前馈神经网络差一点。

Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luck-ily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting.

尽管 CNN 具有引人注目的质量,尽管它们的局部架构相当有效,但将它们大规模的应用到到高分辨率图像中仍然是极其昂贵的。幸运的是,目前的 GPU,搭配了高度优化的 2D 卷积实现,强大到足够促进有趣地大量 CNN 的训练,最近的数据集例如 ImageNet 包含足够的标注样本来训练这样的模型而没有严重的过拟合。

The specific contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly1. Our network contains a number of new and unusual features which improve its performance and reduce its training time, which are detailed in Section 3. The size of our network made overfitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing overfitting, which are described in Section 4. Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than 1% of the model’s parameters) resulted in inferior performance.

本文具体的贡献如下:我们在 ILSVRC-2010 和 ILSVRC-2012[2]的 ImageNet 子集上训练了到目前为止最大的神经网络之一,并取得了迄今为止在这些数据集上报道过的最好结果。我们编写了高度优化的 2D 卷积 GPU 实现以及训练卷积神经网络内部的所有其它操作,我们把它公开了。我们的网络包含许多新的不寻常的特性,这些特性提高了神经网络的性能并减少了训练时间,详见第三节。即使使用了 120 万标注的训练样本,我们的网络尺寸仍然使过拟合成为一个明显的问题,因此我们使用了一些有效的技术来防止过拟合,详见第四节。我们最终的网络包含 5 个卷积层和 3 个全连接层,深度似乎是非常重要的:我们发现移除任何卷积层(每个卷积层包含的参数不超过模型参数的 1%)都会导致更差的性能。

In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.

最后,网络尺寸主要受限于目前 GPU 的内存容量和我们能忍受的训练时间。我们的网络在两个 GTX 580 3GB GPU 上训练五六天。我们的所有实验表明我们的结果可以简单地通过等待更快的 GPU 和更大的可用数据集来提高。

$2 从简介部分可以了解到,本论文主要讲的是 AlexNet 团队使用 GPU 训练了一个大的卷积神经网络,在大数据集中实现了图像分类。并且,在网络中引入了一些新特性,用于提高性能减少训练时间;使用了一些技术防止了过拟合;意识到网络的深度很重要。因此,AlexNet 团队使用到的新特性、防止过拟合技术、GPU 训练是我们需要重点学习的。

2 The Dataset

2 数据集

ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Ama-zon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images.

ImageNet 数据集有超过 1500 万的标注高分辨率图像,这些图像属于大约 22000 个类别。这些图像是从网上收集的,使用了 Amazon’s Mechanical Turk 的众包工具通过人工标注的。从 2010 年起,作为 Pascal 视觉对象挑战赛的一部分,每年都会举办 ImageNet 大规模视觉识别挑战赛(ILSVRC)。ILSVRC 使用 ImageNet 的一个子集,1000 个类别每个类别大约 1000 张图像。总计,大约 120 万训练图像,50000 张验证图像和 15 万测试图像。

ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model.

ILSVRC-2010 是 ILSVRC 竞赛中唯一可以获得测试集标签的版本,因此我们大多数实验都是在这个版本上运行的。由于我们也使用我们的模型参加了 ILSVRC-2012 竞赛,因此在第六节我们也报告了模型在这个版本的数据集上的结果,这个版本的测试标签是不可获得的。在 ImageNet 上,按照惯例报告两个错误率:top-1 和 top-5,top-5 错误率是指测试图像的正确标签不在模型认为的五个最可能的便签之中

ImageNet consists of variable-resolution images, while our system requires a constant input dimen-sionality. Therefore, we down-sampled the images to a fixed resolution of 256 × 256. Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central 256×256 patch from the resulting image. We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels.

ImageNet 包含各种分辨率的图像,而我们的系统要求不变的输入维度。因此,我们将图像进行下采样到固定的 256×256 分辨率。给定一个矩形图像,我们首先缩放图像短边长度为 256,然后从结果图像中裁剪中心的 256×256 大小的图像块。除了在训练集上对像素减去平均活跃度外,我们不对图像做任何其它的预处理。因此我们在原始的 RGB 像素值(中心的)上训练我们的网络。

$3 作者先介绍了 ImageNet 数据集,而后介绍了大多实验使用的 ILSVRC-2010,额外还介绍了 top-5 错误率的概念。重点的是作者对不同分辨率图像的处理,采用的是缩放短边中心裁剪的方法。

3 The Architecture

3 架构

The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first.

我们的网络架构概括为图 2。它包含八个学习层–5 个卷积层和 3 个全连接层。下面,我们将描述我们网络结构中的一些新奇的不寻常的特性。3.1-3.4 小节按照我们对它们评估的重要性进行排序,最重要的最优先

3.1 ReLU Nonlinearity

3.1 ReLU 非线性

The standard way to model a neuron’s output f as a function of its input x is with f(x) = tanh(x) or f(x) = (1 + e−x)−1. In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity f(x) = max(0,x). Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs). Deep convolutional neural net-works with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations re-quired to reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional net-work. This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron models.

将神经元输出 f 建模为输入 x 的函数的标准方式是用 f(x) = tanh(x)f(x) = (1 + e−x)−1。考虑到梯度下降的训练时间,这些饱和的非线性比非饱和非线性 f(x) = max(0,x) 更慢。根据 Nair 和 Hinton[20]的说法,我们将这种非线性神经元称为修正线性单元(ReLU)。采用 ReLU 的深度卷积神经网络训练时间比等价的 tanh 单元要快几倍。在图 1 中,对于一个特定的四层卷积网络,在 CIFAR-10 数据集上达到 25% 的训练误差所需要的迭代次数可以证实这一点。这幅图表明,如果我们采用传统的饱和神经元模型,我们将不能在如此大的神经网络上实验该工作。

Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line). The learning rates for each net-work were chosen independently to make train-ing as fast as possible. No regularization of any kind was employed. The magnitude of the effect demonstrated here varies with network architecture, but networks with ReLUs consis-tently learn several times faster than equivalents with saturating neurons.

图 1:使用 ReLU 的四层卷积神经网络在 CIFAR-10 数据集上达到 25% 的训练误差比使用 tanh 神经元的等价网络(虚线)快 6 倍。为了使训练尽可能快,每个网络的学习率是单独选择的。没有采用任何类型的正则化。影响的大小随着网络结构的变化而变化,这一点已得到证实,但使用 ReLU 的网络都比等价的饱和神经元快几倍

$1 通过图 1 可以了解到使用 ReLU 可以加快训练的速度

We are not the first to consider alternatives to tradi-tional neuron models in CNNs. For example, Jarrett et al. [11] claim that the nonlinearity f (x) = |tanh(x)| works particularly well with their type of contrast nor-malization followed by local average pooling on the Caltech-101 dataset. However, on this dataset the pri-mary concern is preventing overfitting, so the effect they are observing is different from the accelerated ability to fit the training set which we report when us-ing ReLUs. Faster learning has a great influence on the performance of large models trained on large datasets.

我们不是第一个考虑替代 CNN 中传统神经元模型的人。例如,Jarrett 等人[11]声称非线性函数 f(x) = |tanh(x)| 与其对比度归一化一起,然后是局部均值池化,在 Caltech-101 数据集上工作的非常好。然而,在这个数据集上主要的关注点是防止过拟合,因此他们观测到的影响不同于我们使用 ReLU 拟合数据集时的加速能力。更快的学习对大型数据集上大型模型的性能有很大的影响。

$3 作者指出了网络架构图,并重点指出几个新特征,而且按重要程度排序。ReLU 作为第一个,当然是最重要的特性。作者指出,使用非饱和的 ReLu 替代传统的饱和(tanh 等)非线性激活函数,可以加快网络的训练速度,这对于大型数据集的模型性能有很大的帮助。想要详细了解为何非饱和的 ReLU 可以加速训练,可以参考引文 20

3.2 Training on Multiple GPUs

3.2 多 GPU 训练

A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another’s memory directly, without going through host machine memory. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation.

单个 GTX580 GPU 只有 3G 内存,这限制了可以在 GTX580 上进行训练的网络最大尺寸。事实证明 120 万图像用来进行网络训练是足够的,但网络太大因此不能在单个 GPU 上进行训练。因此我们将网络分布在两个 GPU 上。目前的 GPU 非常适合跨 GPU 并行,因为它们可以直接互相读写内存,而不需要通过主机内存。我们采用的并行方案基本上每个 GPU 放置一半的核(或神经元),还有一个额外的技巧只在某些特定的层上进行 GPU 通信。这意味着,例如,第 3 层的核会将第 2 层的所有核映射作为输入。然而,第 4 层的核只将位于相同 GPU 上的第 3 层的核映射作为输入。连接模式的选择是一个交叉验证问题,但这可以让我们准确地调整通信数量,直到它的计算量在可接受的范围内。

The resultant architecture is somewhat similar to that of the “columnar” CNN employed by Cires ̧an et al. [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net2.

除了我们的列不是独立的之外(看图 2),最终的架构有点类似于 Ciresan 等人[5]采用的“columnar” CNN。与每个卷积层一半的核在单 GPU 上训练的网络相比,这个方案降分别低了我们的 top-1 1.7%,top-5 1.2% 的错误率。双 GPU 网络比单 GPU 网络稍微减少了训练时间。

Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities between the two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264– 4096–4096–1000.

图 2:我们的 CNN 架构图解,明确描述了两个 GPU 之间的责任。在图的顶部,一个 GPU 运行在部分层上,而在图的底部,另一个 GPU 运行在部分层上。GPU 只在特定的层进行通信。网络的输入是 150,528 维,网络剩下层的神经元数目分别是 253,440–186,624–64,896–64,896–43,264–4096–4096–1000(8 层)。

$1 图 2 展示了 AlexNet 的网络架构,并说明了多个 GPU 之间如何协作注:这个图只展示了一半内容,原论文就是这样的!!!

$3 作者介绍了如何使用多 GPU 训练模型,来突破网络的最大尺寸。虽然,现在已经有很成熟的分布式训练方案,但是在当时那个时候,这绝对是非常成功的实践。

3.3 Local Response Normalization

3.3 局部响应归一化

ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by the activity of a neuron computed by applying kernel i at position (x, y) and then applying the ReLU nonlinearity, the response-normalized activity is given by the expression

where the sum runs over n “adjacent” kernel maps at the same spatial position, and N is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants k, n, α, and β are hyper-parameters whose values are determined using a validation set; we used k = 2, n = 5, α = 10−4, and β = 0.75. We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5).

ReLU 具有让人满意的特性,它不需要通过输入归一化来防止饱和。如果至少一些训练样本对 ReLU 产生了正输入,那么那个神经元上将发生学习。然而,我们仍然发现接下来的局部响应归一化有助于泛化 表示神经元激活,通过在(x,y)位置应用核 i,然后应用 ReLU 非线性来计算,响应归一化激活 通过下式给定:

求和运算在 n 个“毗邻的”核映射的同一位置上执行,N 是本层的卷积核数目。核映射的顺序当然是任意的,在训练开始前确定。响应归一化的顺序实现了一种侧抑制形式,灵感来自于真实神经元中发现的类型,为使用不同核进行神经元输出计算的较大活动创造了竞争。常量 k,n,α,β 是超参数,它们的值通过验证集确定;我们设 k=2,n=5,α=0.0001,β=0.75。我们在特定的层使用的 ReLU 非线性之后应用了这种归一化(请看 3.5 小节)。

This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a 13% test error rate without normalization and 11% with normalization3.

这个方案与 Jarrett 等人[11]的局部对比度归一化方案有一定的相似性,但我们更恰当的称其为“亮度归一化”,因此我们没有减去均值。响应归一化分别减少了 top-1 1.4%,top-5 1.2%的错误率。我们也在 CIFAR-10 数据集上验证了这个方案的有效性:一个没有归一化的四层 CNN 取得了 13% 的错误率,而使用归一化取得了 11% 的错误率。

$3 作者发现,虽然 ReLU 不需要通过输入归一化来防止饱和,但是局部响应归一化有助于泛化。并且用实际证明,响应归一化可以降低错误率。温馨提示:本次阅读如果没能理解数学公式,可以先跳过。

3.4 Overlapping Pooling

3.4 重叠池化

Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size z × z centered at the location of the pooling unit. If we set s = z, we obtain traditional local pooling as commonly employed in CNNs. If we set s < z, we obtain overlapping pooling. This is what we use throughout our network, with s = 2 and z = 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s = 2, z = 2, which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.

CNN 中的池化层归纳了同一核映射上相邻组神经元的输出。习惯上,相邻池化单元归纳的区域是不重叠的(例如[17, 11, 4])。更确切的说,池化层可看作由池化单元网格组成,网格间距为 s 个像素,每个网格归纳池化单元中心位置 z×z 大小的邻居。如果设置 s=z,我们会得到通常在 CNN 中采用的传统局部池化。如果设置 s<z,我们会得到重叠池化。这就是我们网络中使用的方法,设置 s=2,z=3。这个方案分别降低了 top-1 0.4%,top-5 0.3% 的错误率,与非重叠方案 s=2,z=2 相比,输出的维度是相等的。我们在训练过程中通常观察采用重叠池化的模型,发现它更难过拟合

$3 作者发现使用重叠池化可以降低错误率,并且有避免过拟合的效果

3.5 Overall Architecture

3.5 整体架构

Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fully-connected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution.

现在我们准备描述我们的 CNN 的整体架构。如图 2 所示,我们的网络包含 8 个带权重的层;前 5 层是卷积层,剩下的 3 层是全连接层。最后一层全连接层的输出是 1000 维 softmax 的输入,softmax 会产生 1000 类标签的分布。我们的网络最大化了多项逻辑回归的目标,这等价于最大化预测分布下训练样本正确标签的对数概率的均值。

The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fully- connected layers are connected to all neurons in the previous layer. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer.

第 2,4,5 卷积层的核只与位于同一 GPU 上的前一层的核映射相连接(看图 2)。第 3 卷积层的核与第 2 层的所有核映射相连。全连接层的神经元与前一层的所有神经元相连。第 1,2 卷积层之后是响应归一化层。3.4 节描述的这种最大池化层在响应归一化层和第 5 卷积层之后。ReLU 非线性应用在每个卷积层和全连接层的输出上。

The first convolutional layer filters the 224 × 224 × 3 input image with 96 kernels of size 11 × 11 × 3 with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size 5 × 5 × 48. The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3 × 3 × 256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3 × 3 × 192 , and the fifth convolutional layer has 256 kernels of size 3 × 3 × 192. The fully-connected layers have 4096 neurons each.

第 1 卷积层使用 96 个核对 224 × 224 × 3 的输入图像进行滤波,核大小为 11 × 11 × 3,步长是 4 个像素(核映射中相邻神经元感受野中心之间的距离)。第 2 卷积层使用用第 1 卷积层的输出(响应归一化和池化)作为输入,并使用 256 个核进行滤波,核大小为 5 × 5 × 48。第 3,4,5 卷积层互相连接,中间没有接入池化层或归一化层。第 3 卷积层有 384 个核,核大小为 3 × 3 × 256,与第 2 卷积层的输出(归一化的,池化的)相连。第 4 卷积层有 384 个核,核大小为 3 × 3 × 192,第 5 卷积层有 256 个核,核大小为 3 × 3 × 192。每个全连接层有 4096 个神经元。

$3 这个部分作者主要描述了整体架构,与图 2 描述的一致。

4 Reducing Overfitting

4 减少过拟合

Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting. Below, we describe the two primary ways in which we combat overfitting.

我们的神经网络架构有 6000 万参数。尽管 ILSVRC 的 1000 类使每个训练样本从图像到标签的映射上强加了 10 比特的约束,但这不足以学习这么多的参数而没有相当大的过拟合。下面,我们会描述我们用来克服过拟合的两种主要方式

4.1 Data Augmentation

4.1 数据增强

The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk. In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free.

图像数据上最简单常用的用来减少过拟合的方法是使用标签保留变换(例如[25, 4, 5])来人工增大数据集。我们使用了两种独特的数据增强方式,这两种方式都可以从原始图像通过非常少的计算量产生变换的图像,因此变换图像不需要存储在硬盘上。在我们的实现中,变换图像通过 CPU 的 Python 代码生成而此时 GPU 正在训练前一批图像。因此,实际上这些数据增强方案是计算免费的。

The first form of data augmentation consists of generating image translations and horizontal reflec-tions. We do this by extracting random 224 × 224 patches (and their horizontal reflections) from the 256×256 images and training our network on these extracted patches4. This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly inter-dependent. Without this scheme, our network suffers from substantial overfitting, which would have forced us to use much smaller networks. At test time, the network makes a prediction by extracting five 224 × 224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches.

第一种数据增强方式包括产生图像变换和水平翻转。我们从 256×256 图像上通过随机提取 224 × 224 的图像块实现了这种方式,然后在这些提取的图像块上进行训练。这通过一个 2048 因子增大了我们的训练集,尽管最终的训练样本是高度相关的。没有这个方案,我们的网络会有大量的过拟合,这会迫使我们使用更小的网络。在测试时,网络会提取 5 个 224 × 224 的图像块(四个角上的图像块和中心的图像块)和它们的水平翻转(因此总共 10 个图像块)进行预测,然后对网络在 10 个图像块上的 softmax 层进行平均。

The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components, with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel we add the following quantity:

where pi and λi are ith eigenvector and eigenvalue of the 3 × 3 covariance matrix of RGB pixel values, respectively, and αi is the aforementioned