Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (5): 948-956    DOI: 10.3785/j.issn.1008-973X.2023.05.011
Chip surface character recognition based on convolutional recurrent neural network
Fan XIONG(),Tian CHEN*(),Bai-cheng BIAN,Jun LIU
School of Mechanical Engineering, Shanghai Dianji University, Shanghai 201306, China
Download: HTML     PDF(1721KB) HTML
Export: BibTeX | EndNote (RIS)      


A character recognition method based on an improved convolutional recurrent neural network (CRNN) was proposed for the recognition of characters on the chip surface. The image was binarized by the threshold segmentation based on integral map operation, and the orientation correction of the text field image was completed using affine transformation to achieve the localization of text lines. Based on the original CRNN, the backbone network was replaced with MobileNet-V3 structure and the attention mechanism was added between the two layers of LSTM, while the center loss function was introduced. The improved CRNN was used to implement the text line character recognition and tested on 40 510 chip text line images. The multiple sub-models were obtained by fine-tuning the model training with small sample datasets to achieve integrated inference. The combined recognition accuracy used three models was stable at about 99.97%, and the total recognition time of a single chip image was less than 60 ms. The experimental results showed that the accuracy of the improved CRNN algorithm was improved by about 27.48% over the original CRNN, and the integrated inference of multiple models could achieve higher accuracy.

Key wordsimage processing      integral image      convolutional recurrent neural network      character recognition      integrated inference     
Received: 25 December 2021      Published: 09 May 2023
CLC:  TP 391  
Fund:  上海市地方院校能力建设计划项目(22010501000);上海多向模锻工程技术研究中心资助项目(20DZ2253200)
Corresponding Authors: Tian CHEN     E-mail:;
Cite this article:

Fan XIONG,Tian CHEN,Bai-cheng BIAN,Jun LIU. Chip surface character recognition based on convolutional recurrent neural network. Journal of ZheJiang University (Engineering Science), 2023, 57(5): 948-956.

URL:     OR


基于积分图运算的阈值分割将图像二值化,使用仿射变换完成文本字段图像的方向校正,从而实现文本行的定位.在原始卷积循环神经网络(CRNN)的基础上,将骨干网络替换成MobileNet-V3结构,在2层LSTM之间加入注意力机制,同时引入中心损失函数.利用改进的CRNN实现文本行字符的识别.将改进后的CRNN在40 510 张芯片文本行图像上进行测试.通过小样本数据集进行模型微调训练得到多个子模型,从而实现集成推理,使用3个模型的综合识别准确率稳定在99.97%左右,单张芯片图像的总识别时间小于60 ms.实验结果表明,改进的CRNN算法的准确率比原始CRNN提升了大约27.48%,多模型集成推理的方法可以实现更高的准确率.

关键词: 图像处理,  积分图,  卷积循环神经网络,  字符识别,  集成推理 
Fig.1 Example of integral diagram calculation
Fig.2 Overall scheme process of text line recognition
Fig.3 Original CRNN structure
类型 描述 激活函数 卷积参数
Input W×32 ? ?
Conv Layer 1 3→32 h-swish k: (3,3)s: (2,2)p: (1,1)
Conv Block 1 32→32 relu k:(1,1)-(3,3)-(1,1)s:(1,1)-(1,1)-(1,1)p:(0,0)-(1,1)-(0,0)
Conv Block 2 32→48 relu k:(1,1)-(3,3)-(1,1)s:(1,1)-(2,2)-(1,1)p:(0,0)-(1,1)-(0,0)
Conv Block 3 48→48 relu k:(1,1)-(3,3)-(1,1)s:(1,1)-(1,1)-(1,1)p:(0,0)-(1,1)-(0,0)
Conv Block 4 48→64 h-swish k:(1,1)-(3,3)-(1,1)s:(1,1)-(2,2)-(1,1)p:(0,0)-(1,1)-(0,0)
Conv Block 5 64→96 h-swish k:(1,1)-(3,3)-(1,1)-(1,1)s:(1,1)-(1,1)-(1,1)-(1,1)p:(0,0)-(1,1)-(0,0)-(0,0)
Conv Block 6 96→128 h-swish k:(1,1)-(3,3)-(1,1)-(1,1)s:(1,1)-(1,1)-(1,1)-(1,1)p:(0,0)-(1,1)-(0,0)-(0,0)
Conv Block 7 128→256 h-swish k:(1,1)-(3,3)-(1,1)s:(1,1)-(2,1)-(1,1)p:(0,0)-(1,1)-(0,0)
Conv Block 8 256→256 h-swish k:(1,1)-(3,3)-(1,1)s:(1,1)-(1,1)-(1,1)p:(0,0)-(1,1)-(0,0)
Conv Layer 2 256→512 h-swish k:(2,2) s:(1,1)p:(0,0)
Output 512×1×40 ? ?
Tab.1 Improved CNN module structure
Fig.4 Curve characteristics of swish and h-swish activation functions
Fig.5 Improved LSTM module structure
Fig.6 Multi-model integrated inference
Fig.7 Comparison of threshold segmentation effects of different algorithms
Fig.8 Orientation correction of text field area
Fig.9 Basic dataset images
IC文本行图像 ACC/%
初次训练 二次训练 多模型综合
99.946 99.872 99.981
99.965 99.958 99.973
99.876 99.953 99.963
99.891 99.968 99.976
99.936 99.963 99.973
99.662 97.759 99.847
Tab.2 Integrated inferring accuracy test results
模型形态 A/% T/ms
原始CRNN 67.353 25.04
CNN改进后 74.068 14.58
LSTM改进后 78.474 19.14
损失函数改进后 90.751 23.90
综合改进后 94.831 11.85
Tab.3 Comparative test results of CRNN improvements
[1]   王珂, 杨芳, 姜杉 光学字符识别综述[J]. 计算机应用研究, 2020, 37 (Suppl.2): 22- 24
WANG Ke, YANG Fang, JIANG Shan Overview of optical character recognition[J]. Application Research of Computers, 2020, 37 (Suppl.2): 22- 24
[2]   陈景柱, 鲍玉斌 图像处理中基于改进YOLO的ROI提取算法研究[J]. 数学的实践与认识, 2020, 50 (22): 179- 185
CHEN Jing-zhu, BAO Yu-bin ROI extraction algorithm based on improved YOLO in image processing[J]. Journal of Mathematics in Practice and Theory, 2020, 50 (22): 179- 185
[3]   郭晓峰, 王耀南, 毛建旭 基于几何特征的IC芯片字符分割与识别方法[J]. 智能系统学报, 2020, 15 (1): 144- 151
GUO Xiao-feng, WANG Yao-nan, MAO Jian-xu IC chip character segmentation and recognition method based on geometric features[J]. CAAI Transactions on Intelligent Systems, 2020, 15 (1): 144- 151
doi: 10.11992/tis.201904028
[4]   姚文凤, 甄彤, 吕宗旺, 等 车牌字符分割与识别技术研究[J]. 现代电子技术, 2020, 43 (19): 65- 69
YAO Wen-feng, ZHEN Tong, LV Zong-wang, et al Research on technology of segmentation and recognition of license plate character[J]. Modern Electronics Technique, 2020, 43 (19): 65- 69
doi: 10.16652/j.issn.1004-373x.2020.19.016
[5]   马欣欣, 李小平 集装箱箱号字符识别关键技术的研究[J]. 现代电子技术, 2019, 42 (14): 131- 134
MA Xin-xin, LI Xiao-ping Research on key technologies for character recognition of container numbers[J]. Modern Electronics Technique, 2019, 42 (14): 131- 134
doi: 10.16652/j.issn.1004-373x.2019.14.030
[6]   白睿, 徐友春, 李永乐, 等 智能车道路场景数字字符识别技术[J]. 激光与光电子学进展, 2021, 57 (15): 178- 185
BAI Rui, XU You-chun, LI Yong-le, et al Digital character recognition technique for intelligent vehicles in road scenes[J]. Laser and Optoelectronics Progress, 2021, 57 (15): 178- 185
[7]   祁忠琪, 涂凯, 吴书楷, 等 基于深度学习的含堆叠字符的车牌识别算法[J]. 计算机应用研究, 2021, 38 (5): 1550- 1554
QI Zhong-qi, TU Kai, WU Shu-kai, et al Recognizing license plate with stacked characters based on deep learning[J]. Application Research of Computers, 2021, 38 (5): 1550- 1554
doi: 10.19734/j.issn.1001-3695.2020.04.0147
[8]   VISHNUVARDHAN A, SRIHARSHA M N An overview of text detection in natural scene images[J]. International Journal of Innovative Technology and Exploring Engineering, 2019, 8 (7c2): 384- 387
[9]   ZHAI W, GAO T, FENG J Research on pre-processing methods for license plate recognition[J]. International Journal of Computer Vision and Image Processing, 2021, 11 (1): 47- 79
doi: 10.4018/IJCVIP.2021010104
[10]   LAROCA R, ZANLORENSI L A, GONÇALVES G R, et al An efficient and layout-independent automatic license plate recognition system based on the YOLO detector[J]. IET Intelligent Transport Systems, 2021, 15 (4): 483- 503
doi: 10.1049/itr2.12030
[11]   CHEN Z, YAN L, YIN S, et al Vehicle license plate recognition system based on deep learning in natural scene[J]. Journal of Artificial Intelligence, 2020, 2 (4): 167
doi: 10.32604/jai.2020.012716
[12]   NAIEMI F, GHODS V, KHALESI H A novel pipeline framework for multi oriented scene text image detection and recognition[J]. Expert Systems with Applications, 2021, 170: 114549
doi: 10.1016/j.eswa.2020.114549
[13]   HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[14]   殷航. 基于YOLO的自然场景中文文本检测算法研究[D]. 武汉: 武汉科技大学, 2020: 17.
YIN Hang. Research on Chinese text detection algorithm for natural scenes based on YOLO [D]. Wuhan: Wuhan University of Science and Technology, 2020: 17.
[15]   REDMON J, FARHADI A. YOLOv3: an incremental improvement [EB/OL]. [2019-02-25].
[16]   傅勇, 潘晴, 田妮莉, 等 改进级联卷积神经网络的平面旋转人脸检测[J]. 计算机工程与设计, 2020, 41 (3): 856- 861
FU Yong, PAN Qing, TIAN Ni-li, et al Face detection of rotation in plane based on improved cascade CNN[J]. Computer Engineering and Applications, 2020, 41 (3): 856- 861
doi: 10.16208/j.issn1000-7024.2020.03.041
[17]   SHI B G, BAI X, YAO C An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39 (11): 2298- 2304
[18]   MERZBAN M H, MAHMOUD E Efficient solution of Otsu multilevel image thresholding: a comparative study[J]. Expert Systems with Applications, 2019, 116: 299- 309
doi: 10.1016/j.eswa.2018.09.008
[19]   BRADLEY D, ROTH G Adaptive thresholding using the integral image[J]. Journal of Graphics Tools, 2007, 12 (2): 13- 21
doi: 10.1080/2151237X.2007.10129236
[20]   SHI X, CHEN Z, WANG H, et al. Convolutional LSTM network: a machine learning approach for precipitation nowcasting [C]// Advances in Neural Information Processing Systems. Montreal: NIPS, 2015: 802-810.
[21]   SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition [C]// International Conference on Learning Representations. San Diego: ICLR, 2015: 1-14.
[22]   HOWARD A, SANGLER M, CHU G, et al. Searching for mobilenetv3 [C]// Proceedings of the IEEE/CVF Intern-ational Conference on Computer Vision. California: IEEE/CVF, 2019: 1314-1324.
[23]   BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate [C]// International Conference on Learning Representations. San Diego: ICLR, 2015: 1-15. .
[1] Pei-zhi WEN,Jun-mou CHEN,Yan-nan XIAO,Ya-yuan WEN,Wen-ming HUANG. Underwater image enhancement algorithm based on GAN and multi-level wavelet CNN[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 213-224.
[2] Hao JIANG,Hai-song XU. Histogram based tone mapping algorithm using image segmentation and fusion[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2224-2231.
[3] Tong CHEN,Jian-feng GUO,Xin-zhong HAN,Xue-li XIE,Jian-xiang XI. Visible and infrared image matching method based on generative adversarial model[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(1): 63-74.
[4] Zhu-ye XU,Xiao-qiang ZHAO,Hong-mei JIANG. 3D model fitting method based on point distribution model[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(12): 2373-2381.
[5] Ying LI,Fang CHENG,Zhi-lin ZHAO. Machining precision online measurement of large span pin hole using structured light[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(3): 557-565.
[6] Wan-liang WANG,Xiao-han YANG,Yan-wei ZHAO,Nan GAO,Chuang LV,Zhao-juan ZHANG. Image enhancement algorithm with convolutional auto-encoder network[J]. Journal of ZheJiang University (Engineering Science), 2019, 53(9): 1728-1740.
[7] ZHOU Hao, LI Ning, LI Yuan, ZHAO Meng-hao, CEN Ke-fa. Experimental study on ethanol spray combustion characteristics under oxy-fuel conditions[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(9): 1821-1827.
[8] ZHANG Cheng-zhi, FENG Hua-jun, XU Zhi-hai, LI Qi, CHEN Yue-ting. Piecewise noise variance estimation of images based on wavelet transform[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(9): 1804-1810.
[9] ZHOU Jia-li, CHEN Yi-jun, WU Min. Image acquisition and preprocessing method based on FPGA monitor[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(2): 398-405.
[10] HUANG Song, YI Ben-shun. Underwater image restoration algorithm based on adaptive transmission ratio[J]. Journal of ZheJiang University (Engineering Science), 2018, 52(1): 166-173.
[11] ZHOU Hao, MA Wei chen, YANG Yu, CHEN Jian zhong. Investigation of flame characteristics of low NOx swirl burner[J]. Journal of ZheJiang University (Engineering Science), 2016, 50(4): 698-703.
[12] WU Yi,YIN Jun, ZHU Li, YUAN Yong ming. Two dimensional Arimoto gray entropy image thresholding based on bee colony optimization or decomposition[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(9): 1625-1633.
[13] WANG Yuan-yuan, GUO Yan-en, SHI Guo-quan, WEI Jun-xia, XIA Shun-ren. Algorithm for seabed terrain boundary extraction based on acoustic images[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(2): 376-383.
[14] WANG Han, XIA Xin-xing, YU Chao, ZHONG Qing, GAO Qian-jue, LI Hai-feng, LIU Xu. Brightness uniformity correction method of light field integral three-dimensional display[J]. Journal of ZheJiang University (Engineering Science), 2015, 49(1): 1-5.
[15] WANG Han, XIA Xin-xing, YU Chao, ZHONG Qing, GAO Qian-jue, LI Hai-feng, LIU Xu. integral three-dimensional display[J]. Journal of ZheJiang University (Engineering Science), 2014, 48(8): 1-6.