Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2019, Vol. 53 Issue (2): 373-381    DOI: 10.3785/j.issn.1008-973X.2019.02.021
Computer and Control Engineering     
Attention Res-Unet: an efficient shadow detection algorithm
Yue DONG(),Hua-jun FENG(),Zhi-hai XU,Yue-ting CHEN,Qi LI*()
State Key Laboratory of Optical Instrumentation, Zhejiang University, Hangzhou 310027, China
Download: HTML     PDF(1545KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

Shadow pixels in images can lead to the uncertainty of image content, which is harmful to computer vision tasks. Therefore, shadow detection is often used as a preprocessing step of computer vision algorithm. A shadow detection network was proposed by combining semantic information contained in input images and correlation between pixels. Pre-trained deep network ResNeXt101 was used as feature extraction front-end module to extract semantic information of the image. The baseline structure of the network was built to up-sample feature layers, encouraged by the design idea of U-Net. Non-local operations were added before the output layer to provide global information for each pixel and establish the relationship between pixels. At the same time, an attention generation module and an attention fusion module were developed to further improve shadow detection accuracy. Two common shadow detection datasets named SBU and UCF were utilized for verification. Experiment results showed that the proposed network outperformed previous methods in both visual effect and objective indicator. The proposed network showed 14.4% reduction on SBU and 14.9% reduction on UCF for the balance error rate, compared with the state-of-the-art framework.



Key wordsshadow detection      feature extraction      semantic information      pixel correlation      non-local      attention module      convolutional neural network (CNN)     
Received: 14 October 2018      Published: 21 February 2019
CLC:  TP 391  
Corresponding Authors: Qi LI     E-mail: 21730058@zju.edu.cn;fenghj@zju.edu.cn;liqi@zju.edu.cn
Cite this article:

Yue DONG,Hua-jun FENG,Zhi-hai XU,Yue-ting CHEN,Qi LI. Attention Res-Unet: an efficient shadow detection algorithm. Journal of ZheJiang University (Engineering Science), 2019, 53(2): 373-381.

URL:

http://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2019.02.021     OR     http://www.zjujournals.com/eng/Y2019/V53/I2/373


Attention Res-Unet: 一种高效阴影检测算法

图像中阴影像素的存在会导致图像内容的不确定性,对计算机视觉任务有害,因此常将阴影检测作为计算机视觉算法的预处理步骤. 提出全新的阴影检测网络结构,通过结合输入图像中包含的语义信息和像素之间的关联,提升网络性能. 使用预训练后的深层网络ResNeXt101作为特征提取前端,提取图像的语义信息,并结合U-net的设计思路,搭建网络结构,完成特征层的上采样过程. 在输出层之前使用非局部操作,为每一个像素提供全局信息,建立像素与像素之间的联系. 设计注意力生成模块和注意力融合模块,进一步提高检测准确率. 分别在SBU、UCF这2个阴影检测数据集上进行验证,实验结果表明,所提方法的目视效果及客观指标皆优于此前最优方法所得结果,在2个数据集上的平均检测错误率分别降低14.4%和14.9%.


关键词: 阴影检测,  特征提取,  语义信息,  像素关联,  非局部操作,  注意力机制,  卷积神经网络(CNN) 
Fig.1 Shadow origin schematic diagram
Fig.2 Example for hard shadow detection
Fig.3 Network architecture of Attention Res-Unet algorithm
Fig.4 Procedure scheme of attention generation module in Attention Res-Unet algorithm
Fig.5 Procedure scheme of attention fusion module in Attention Res-Unet algorithm
Fig.6 Visual effect comparison of test results of Attention Res-Unet with scGAN[17], patched-CNN[16] and stacked-CNN[15]
方法 SBU[15] UCF[23]
BER SER NER BER SER NER
Attention Res-Unet 4.88 5.31 4.44 8.42 8.40 8.44
A+D-Net[18] 5.70 6.20 5.20 9.90 7.30 12.50
scGAN[17] 9.10 7.80 10.40 11.50 7.70 15.30
patched-CNN[16] 11.20 10.13 12.27
stacked-CNN[15] 11.00 9.60 12.50 13.00 9.00 17.10
Tab.1 Quantitative evaluation results of Attention Res-Unet, A+D-Net[18], scGAN[17], patched-CNN[16] and stacked-CNN[15]
方法 SBU[15]
BER SER NER
U-net 7.69 10.48 4.90
Res-Unet 6.56 7.40 5.73
Res-Unet+Attenion 5.33 5.88 4.77
Attention Res-Unet 4.88 5.31 4.44
Tab.2 Effect validation of each module in Attention Res-Unet
Fig.7 Visual effect comparison of test results of Attention Res-Unet with Res-Unet+Attenion, Res-Unet and U-net
Fig.8 Other results produced by Attention Res-Unet
Fig.9 Failure cases of Attention Res-Unet
[1]   MURALI S, GOVINDAN V K, KALADY S A survey on shadow detection techniques in a single image[J]. Information Technology and Control, 2018, 47 (1): 75- 92
[2]   FINLAYSON G D, DREW M S, LU C. Intrinsic images by entropy minimization [C]// Computer Vision-ECCV. Prague: Springer, 2004: 582–595.
[3]   FINLAYSON G D, HORDLEY S D, LU C On the removal of shadows from images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28 (1): 59- 68
[4]   FINLAYSON G D, DREW M S, LU C Entropy minimization for shadow removal[J]. International Journal of Computer Vision, 2009, 85 (1): 35- 57
[5]   HOIEM D. Single-image shadow detection and removal using paired regions [C] // Computer Vision and Pattern Recognition. Colorado Springs: IEEE, 2011: 2033–2040.
[6]   GUO R, DAI Q, HOIEM D Paired regions for shadow detection and removal[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35 (12): 2956- 2967
[7]   VICENTE T F Y, YU C P, SAMARAS D. Single image shadow detection using multiple cues in a supermodular MRF [C] // British Machine Vision Conference. Bristol: British Machine Vision Association, 2013: 1–12.
[8]   YUAN X, EBNER M, WANG Z Single-image shadow detection and removal using local colour constancy computation[J]. Image Processing Iet, 2015, 9 (2): 118- 126
[9]   PANAGOPOULOS A, WANG C, SAMARAS D. Estimating shadows with the bright channel cue [C] // European Conference on Trends and Topics in Computer. Heraklion: Springer, 2010: 1–12.
[10]   MARYAM G, FATIMAH K, ABDULLAH L N Shadow detection using color and edge information[J]. Journal of Computer Science, 2013, 9 (11): 1575- 1588
[11]   SALVADOR E, CAVALLARO A, EBRAHIMI T. Cast shadow segmentation using invariant color features [J]. Computer Vision and Image Understanding, 2004, 95(2): 238-259.
[12]   DONG Q, LIU Y, ZHAO Q Detecting soft shadows in a single outdoor image: from local edge-based models to global constraints[J]. Computers and Graphics, 2014, 38 (1): 310- 319
[13]   TIAN J, QI X, QU L New spectrum ratio properties and features for shadow detection[J]. Pattern Recognition, 2016, 51 (C): 85- 96
[14]   KHAN S H, BENNAMOUN M, SOHEL F. Automatic feature learning for robust shadow detection [C] // Computer Vision and Pattern Recognition. Columbus: IEEE, 2014: 1939–1946.
[15]   VICENTE T F Y, HOU L, YU C P. Large-scale training of shadow detectors with noisily-annotated shadow examples [C] // Computer Vision-ECCV. Amsterdam: Springer, 2016: 816–832.
[16]   HOSSEINZADEH S, SHAKERI M, ZHANG H. Fast shadow detection from a single image using a patched convolutional neural network [EB/OL]. (2018-03-16)[2018-10-14]. http.//arxiv.org/abs/1709.09283.
[17]   NGUYEN V, VICENTE T F Y, ZHAO M. Shadow detection with conditional generative adversarial networks [C] // IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 4520–4528.
[18]   LE H, VICENTE T F Y, NGUYEN V. A+D-Net: shadow detection with adversarial shadow attenuation. [EB/OL]. (2018-07-27)[2018-10-14]. https://arxiv.org/abs/1712.01361.
[19]   XIE S, GIRSHICK R, DOLLAR P. Aggregated residual transformations for deep neural networks [C]// Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5987–5995.
[20]   RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation [C] // International Conference on Medical Image Computing and Computer-assisted Intervention. Cham: Springer, 2015: 234–241.
[21]   WANG F, JIANG M, QIAN C. Residual attention network for image classification [C] // Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 6450–6458.
[22]   WANG X, GIRSHICK R, GUPTA A. Non-local neural networks [EB/OL]. (2018-04-18)[2018-10-14]. https://arxiv.org/abs/1711.07971.
[23]   ZHU J, SAMUEL K G G, MASOOD S Z. Learning to recognize shadows in monochromatic natural images [C] // Computer Vision and Pattern Recognition. San Francisco: IEEE, 2010: 223–230.
[24]   IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift [C]// International Conference on Machine Learning. Lille: ACM, 2015: 448-456.
[1] Fang LIU,Zhen WANG,Rui-di LIU,Kai WANG. Short-term forecasting method of wind power generation based on BP neural network with combined loss function[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(3): 594-600.
[2] Qiao-hong CHEN,YI CHEN,Wen-shu Li,Yu-bo JIA. Clothing image classification based on multi-scale SE-Xception[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(9): 1727-1735.
[3] Ping YANG,Dan WANG,Zi-jian KAGN,Tong LI,Li-hua FU,Yue-ren YU. Prediction model of paroxysmal atrial fibrillation based on pattern recognition and ensemble CNN-LSTM[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(5): 1039-1048.
[4] Yi-xiong FENG,Kang-jie LI,Yi-cong GAO,Hao ZHEN,Jian-rong TAN. Shaft surface defect detection method based on feature and morphology reconstruction[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 427-434.
[5] Mei-ying QIAO,Xia-xia TANG,Shu-hao YAN,Jian-ke SHI. Bearing fault diagnosis based on improved sparse filter and deep network fusion[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(12): 2301-2309.
[6] Zi-yu JIA,You-fang LIN,Hong-jun ZHANG,Jing WANG. Sleep stage classification model based ondeep convolutional neural network[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1899-1905.
[7] Hong-guang LI,Ying GUO,Ping SUI,Zi-sen QI. Frequency hopping modulation recognition of convolutional neural network based on time-frequency characteristics[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1945-1954.
[8] Gang YE,Yi-bo LI,Zhu-xi MA,Jie CHENG. End-to-end aluminum strip surface defects detection and recognition method based on ViBe[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(10): 1906-1914.
[9] Xiao-hu ZHAO,Liang-fei YIN,Cheng-long ZHAO. Image captioning based on global-local feature and adaptive-attention[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(1): 126-134.
[10] Yan LV,Meng ZHANG,Wu-hao JIANG,Yi-hua NI,Xiao-hong QIAN. Design of elderly fall detection system using CNN[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(6): 1130-1138.
[11] Gui-ran HE,Qi LI,Hua-jun FENG,Zhi-hai XU,Yue-ting CHEN. Dual-focal camera continuous digital zoom based onCNN and feature extraction[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(6): 1182-1189.
[12] Jing-chang WANG,Ling CHEN,Shan-shan YU,Chen-shu JIANG,Yong WU. Multi-factor perceived short-term tourist number prediction model based on gated recurrent unit[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(12): 2357-2364.
[13] Xiang-long ZHAO,Jie CHEN,Rong-jing HONG,Hua WANG,Yuan-yuan LI. Adaptive feature extraction method for slewing bearing based on Wavelet leader and optimized isometric mapping method[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(11): 2092-2101.
[14] Hao LI,Wen-jie ZHAO,Bo HAN. Convolutional neural network acceleration algorithm based on filters pruning[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(10): 1994-2002.
[15] TONG Shui-guang, ZHANG Yi-dong, XU Jian, CONG Fei-yun. Spectral band refined composite multiscale fuzzy entropy and its application in fault diagnosis of rolling bearings[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(8): 1509-1516.