| Automatic Technology, Computer Technology |
|
|
|
|
| Design of activation function in CNN for image classification |
Hong-xia WANG( ),Jia-qi ZHOU,Cheng-hao GU,Hong LIN*() |
| School of Computer Science and Technology, Wuhan University of Technology, Wuhan 430063, China |
|
|
|
Abstract A new combinatorial activation function called relu-softsign was proposed aiming at the problem that the derivative of the commonly used activation function relu in the convolutional neural network is constant to zero at the x negative axis, which makes it easy to cause neuron necrosis during training, and the existing combinatorial activation function relu-softplus can only use the small learning rate in the case of model convergence, which leads to slow convergence. The image classification effect was improved. The role of the activation function during training was analyzed, and the key points that need to be considered in the design of the activation function were given. The relu and softsign functions were combined piecewise in the positive and negative semi axis of the x axis according to these points, so that the derivative of x negative semi axis was no longer constant to zero. Then comparision with the single activation function and relu-softplus combination activation function was conducted on the MNIST, PI100, CIFAR-100 and Caltech256 datasets. The experimental results show that the combinatorial activation function relu-softsign improves the model classification accuracy, simply and effectively mitigates the irreversible " necrosis” phenomenon of neurons. The convergence speed of the model is accelerated, especially on complex data sets.
|
|
Received: 31 October 2018
Published: 25 June 2019
|
|
|
|
Corresponding Authors:
Hong LIN
E-mail: 99575522@qq.com;linhong@whut.edu.cn
|
用于图像分类的卷积神经网络中激活函数的设计
为了提高图像分类效果，针对卷积神经网络中常用激活函数relu在x负半轴的导数恒为零，导致训练过程中容易造成神经元“坏死”以及现有组合激活函数relu-softplus在模型收敛情况下学习率过小导致收敛速度慢的问题，提出新的组合激活函数relu-softsign. 分析激活函数在训练过程中的作用，给出激活函数在设计时需要考虑的要点；根据这些要点，将relu和softsign函数于x轴正、负半轴进行分段组合，使其x负半轴导数不再恒为零；分别在MNIST、PI100、CIFAR-100和Caltech256数据集上，与单一的激活函数和relu-softplus组合激活函数进行对比实验. 实验结果表明，使用relu-softsign组合激活函数提高了模型分类准确率，简单有效地缓解了神经元不可逆“坏死”现象；加快了模型的收敛速度，在复杂数据集上该组合函数的收敛性能更好.
关键词：
图像分类,
卷积神经网络,
激活函数,
relu,
神经元坏死,
组合激活函数
|
|
| [1] |
黄凯奇, 任伟强, 谭铁牛 图像物体分类与检测算法综述[J]. 计算机学报, 2014, 36 (6): 1225- 1240
HUANG Kai-qi, REN Wei-qiang, TAN Tie-niu A review on image object classification and detection[J]. Chinese Journal of Computers, 2014, 36 (6): 1225- 1240
|
|
|
| [2] |
常亮, 邓小明, 周明全, 等 图像理解中的卷积神经网络[J]. 自动化学报, 2016, 42 (9): 1300- 1312
CHANG Liang, DENG Xiao-ming, ZHOU Ming-quan, et al Convolution neural network in image understanding[J]. Acta Automatica Sinica, 2016, 42 (9): 1300- 1312
|
|
|
| [3] |
吴正文. 卷积神经网络在图像分类中的应用研究[D]. 成都: 电子科技大学, 2015.
WU Zheng-wen. Application of convolution neural network in image classification [D]. Chengdu: University of Electronic Science and Technology of China, 2015.
|
|
|
| [4] |
KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks [C] // International Conference on Neural Information Processing Systems. Lake Tahoe: Springer, 2012: 1097-1105.
|
|
|
| [5] |
NAIR V, HINTON G E. Rectified linear units improve restricted Boltzmann machines [C] // Proceedings of the 27th International Conference on Machine Learning (ICML-10). Haifa: Omnipress, 2010: 807-814.
|
|
|
| [6] |
DOLEZEL P, SKRABANEK P, GAGO L Weight initialization possibilities for feedforward neural network with linear saturated activation functions[J]. IFAC-PapersOnLine, 2016, 49 (25): 49- 54
doi: 10.1016/j.ifacol.2016.12.009
|
|
|
| [7] |
MAAS A L, HANNUN A Y, NG A Y. Rectifier nonlinearities improve neural network acoustic models [C] // Proceedings of the 30th International Conference on Machine Learning. Atlanta: ACM, 2013: 456-462.
|
|
|
| [8] |
CLEVERT D A, UNTERTHINER T, HOCHREITER S Fast and accurate deep network learning by exponential linear units (ELUs)[J]. Computer Science, 2015, 5 (2): 716- 730
|
|
|
| [9] |
HE K, ZHANG X, REN S, et al. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification [C] // Proceedings of the IEEE international conference on computer vision. Santiago: IEEE, 2015: 1026-1034.
|
|
|
| [10] |
石琪. 基于卷积神经网络图像分类优化算法的研究与验证[D]. 北京: 北京交通大学, 2017.
SHI Qi. Research and verification of image classification optimization algorithm based on convolutional neural network [D]. Beijing: Beijing Jiaotong University, 2017.
|
|
|
| [11] |
LECUN Y, BOTTOU L, BENGIO Y, et al Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86 (11): 2278- 2324
doi: 10.1109/5.726791
|
|
|
| [12] |
Microsoft Research: product image categorization data set (PI 100)[DB/OL]. [2010-11-01]. http://research.microsoft.com/en-us/people/xingx/pi100.aspx.
|
|
|
| [13] |
FERRARI V, JURIE F, SCHMID C From images to shape models for object detection[J]. International Journal of Computer Vision, 2010, 87 (3): 284- 303
doi: 10.1007/s11263-009-0270-9
|
|
|
| [14] |
GRIFFIN G, HOULUB A, PERONA P. The Caltech-256. Technical report [R]. Pasadena: California Institute of Technology, 2007.
|
|
|
| [15] |
李明威. 图像分类中的卷积神经网络方法研究 [D]. 南京: 南京邮电大学, 2016.
LI Ming-wei. Research of convolutional neural network in image classification [D]. Nanjing: Nanjing University of Posts and Telecommunications, 2016.
|
|
|
| [16] |
DUDA R O, HART P E, STORK D G. Pattern classification [M]. [S.l.]: Wiley, 2004.
|
|
|
| [17] |
贾世杰. 基于内容的商品图像分类方法研究[D]. 大连: 大连理工大学, 2013.
JIA Shi-jie. Research on content based classification of commodity image [D]. Dalian: Dalian University of Technology, 2013.
|
|
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
