Please wait a minute...
浙江大学学报(工学版)
Journal of ZheJiang University (Engineering Science)  2022, Vol. 56 Issue (3): 510-519, 530    DOI: 10.3785/j.issn.1008-973X.2022.03.010
    
Review of Chinese font style transfer research based on deep learning
Ruo-ran CHENG(),Xiao-li ZHAO*(),Hao-jun ZHOU,Han-chen YE
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China
Download: HTML     PDF(874KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

The research works of Chinese font style transfer were classified according to different stages of research development. The traditional methods were briefly reviewed and the deep learning-based methods were combed and analyzed. The commonly used open data sets and evaluation criteria were introduced. The future research trends were expected from four aspects, which were to improve the generation quality, enhance personalized differences, reduce the number of training samples, and learn calligraphy font style.

Key words： font style transfer      deep learning      image translation      neural network      font generation     
Received: 26 May 2021      Published: 29 March 2022
CLC:  TP 391  
Fund:  国家自然科学基金资助项目（61772328）
Corresponding Authors: Xiao-li ZHAO     E-mail: 492409525@qq.com;evawhy@163.com
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Ruo-ran CHENG
Xiao-li ZHAO
Hao-jun ZHOU
Han-chen YE
Cite this article:

Ruo-ran CHENG,Xiao-li ZHAO,Hao-jun ZHOU,Han-chen YE. Review of Chinese font style transfer research based on deep learning. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 510-519, 530.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2022.03.010     OR     https://www.zjujournals.com/eng/Y2022/V56/I3/510


基于深度学习的中文字体风格转换研究综述

根据中文字体风格转换研究发展的不同阶段进行方法分类，简要回顾传统方法，梳理分析深度学习方法. 介绍常用的公开数据集和评价标准. 分别从提高生成质量、增强个性化差异、减少训练样本数量和学习书法字体风格共4个方面展望未来研究.


关键词： 字体风格转换,  深度学习,  图像翻译,  神经网络,  字体生成 
Fig.1 Important research work in different stages of Chinese font style transfer
网络层 图像尺寸/卷积核尺寸 网络层 图像尺寸/卷积核尺寸
输入图像 160 × 160 卷积层 × n 7 × 7 × 128
卷积层 × 2 64 × 64 × 8 卷积层 × 2 3 × 3 × 128
卷积层 × n 32 × 32 × 32 最大池化 2 × 2
卷积层 × n 16 × 16 × 64 输出图像 80 × 80
Tab.1 Network Architect of Rewrite
Fig.2 Samples of fonts for web-accessible datasets
Fig.3 Data set and experimental results of deep learning method（generated characters on top，target characters on bottom）
[1]   STRASSMANN S. Hairy brushes [C]// Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 1986: 225-232.
[2]   LEE J Physically-based modeling of brush painting[J]. Computer Networks and ISDN Systems, 1997, 29 (14): 1571- 1576
doi: 10.1016/S0169-7552(97)00073-1
[3]   LEE J Simulating oriental black-ink painting[J]. IEEE Computer Graphics Application, 1999, 19 (3): 74- 81
doi: 10.1109/38.761553
[4]   张宪荣 电子计算机毛笔字体的形成[J]. 计算机工程, 1985, (3): 40- 45
ZHANG Xian-rong Generation of Chinese brush character font by computer[J]. Computer Engineering, 1985, (3): 40- 45
[5]   陈实午 一种由点阵汉字生成高质量矢量汉字的方法[J]. 计算机研究与发展, 1989, (8): 47- 50
CHEN Shi-wu A method for generating high-quality vector Chinese characters from dot matrix Chinese characters[J]. Journal of Computer Research and Development, 1989, (8): 47- 50
[6]   王新 点阵字库自动生成曲线字库[J]. 计算机应用, 1995, (1): 61- 62
WANG Xin Building curvilinear character-library from dots character-library automatically[J]. Journal of Computer Applications, 1995, (1): 61- 62
[7]   GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 2414-2423.
[8]   YANN L, BOTTOU L Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86 (11): 2278- 2324
doi: 10.1109/5.726791
[9]   GOODFELLOW I, POUGETABADIE J, MIRZA M, et al Generative adversarial networks[J]. Advances in Neural Information Processing Systems, 2014, (3): 2672- 2680
[10]   JOHNSON J, ALAHI A, LI F F Perceptual losses for real-time style transfer and super-resolution[J]. Lecture Notes in Computer Science, 2016, 9906: 694- 711
[11]   TIAN Y. Rewrite [CP/OL]. [2021-05-26]. https://github.com/kaonashi-tyc/Rewrite.
[12]   JIANG Y, LIAN Z H, TANG Y M, et al. DCFont: an end-to-end deep Chinese font generation system [C]// SIGGRAPH Asia 2017 Technical Briefs. New York: ACM, 2017: 1–4.
[13]   CHUA Y S Bézier brushstrokes[J]. Computer-Aided Design, 1990, 22 (9): 550- 555
doi: 10.1016/0010-4485(90)90040-J
[14]   HSU S C, LEE I. Drawing and animation using skeletal strokes [C]// Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 1994: 109-118.
[15]   于金辉, 张积东, 丛延奇 一个基于骨架的笔刷模型[J]. 计算机辅助设计与图形学学报, 1996, (4): 2- 6
YU Jin-hui, ZHANG Ji-dong, CONG Yan-qi A physically based brush pen model[J]. Journal of Computer-Aided Design and Computer Graphics, 1996, (4): 2- 6
[16]   宓晓峰, 唐敏, 林建贞, 等 基于经验的虚拟毛笔模型[J]. 计算机研究与发展, 2003, (8): 1244- 1251
MI Xiao-feng, TANG Min, LIN Jian-zhen, et al An experience based virtual brush model[J]. Journal of Computer Research and Development, 2003, (8): 1244- 1251
[17]   IP H H S, WONG H T F. Calligraphic character synthesis using a brush model [C]// Proceedings Computer Graphics International. [S. l.]: IEEE, 1997: 13-21.
[18]   WONG H T F, IP H H S Virtual brush: a model-based synthesis of Chinese calligraphy[J]. Computer Graphics, 2000, 24 (1): 99- 113
doi: 10.1016/S0097-8493(99)00141-7
[19]   朱墨子, 申飞, 吴仲成, 等 基于统计分析的虚拟毛笔模型及其应用[J]. 计算机工程, 2009, 35 (16): 283- 284
ZHU Mo-zi, SHEN Fei, WU Zhong-cheng, et al Virtual brush model based on statistical analysis and its application[J]. Computer Engineering, 2009, 35 (16): 283- 284
doi: 10.3969/j.issn.1000-3428.2009.16.102
[20]   YHANG Y C, LIU Y, ZHU Q. Modeling virtual Chinese brush using camshaft curve and stochastic process [C]// International Conference on Computer-aided Design and Computer Graphics. Piscataway: IEEE, 2015: 228-229.
[21]   BAXTER B, SCHEIB V, LIN M C, et al. DAB: interactive haptic painting with 3D virtual brushes [C]// Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 2001: 461-468.
[22]   BAXTER W, LIN M C. A versatile interactive 3D brush model [C]// Proceedings of the 12th Pacific Conference on Computer Graphics and Applications. New York: IEEE, 2004: 319-328.
[23]   ADAMS B, WICKE M, DUTRE P, et al. Interactive 3D painting on point-sampled objects [C]// Proceedings of the 1st Eurographics Conference on Point-Based Graphics. Goslar: Eurographics Association, 2004: 57-66.
[24]   CHU N S H, TAI C L. An efficient brush model for physically-based 3D painting [C]// Proceedings of the 10th Pacific Conference on Computer Graphics and Applications. Beijing: IEEE, 2002: 413-421.
[25]   XU S H, TANG M, LAU F, et al A solid model based virtual hairy brush[J]. Computer Graphics Forum, 2002, 21 (3): 299- 308
doi: 10.1111/1467-8659.00589
[26]   XU S H, LAU F, TANG F, et al Advanced design for a realistic virtual brush[J]. Computer Graphics Forum, 2003, 22 (3): 533- 542
doi: 10.1111/1467-8659.t01-2-00701
[27]   BAXTER W, GOVINDARAJU N. Simple data-driven modeling of brushes [C]// Proceedings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. New York: ACM, 2010: 135-142.
[28]   GUO C, HOU Z X, SHI Y Z, et al A virtual 3D interactive painting method for Chinese calligraphy and painting based on real-time force feedback technology[J]. Frontiers of Information Technology and Electronic Engineering, 2017, 18 (11): 1843- 1853
doi: 10.1631/FITEE.1601283
[29]   高晓, 蔡士杰 一种从黑体到隶书的汉字字形自动变体方法[J]. 软件学报, 1995, (9): 542- 550
GAO Xiao, CAI Shi-jie An automatic derivation of Chinese characters typeface from Hei-Ti to Li-Shu[J]. Journal of Software, 1995, (9): 542- 550
[30]   庞健雄, 冯著明 曲线轮廓汉字自动生成技术研究[J]. 桂林电子工业学院学报, 1996, (4): 50- 55
PANG Jian-xiong, FENG Zhu-ming Research of automatic generation for curve outline Chinese font[J]. Journal of Guilin University of Electronic Technology, 1996, (4): 50- 55
[31]   潘志庚, 马小虎, 石教英 动态汉字库自动生成算法[J]. 自动化学报, 1996, (5): 561- 567
PAN Zhi-geng, MA Xiao-hu, SHI Jiao-ying The automatic generation algorithm for dynamic Chinese font[J]. Acta Automatica Sinica, 1996, (5): 561- 567
[32]   潘志庚, 马小虎, 张明敏, 等 基于Fourier级数描述器的多种汉字字形自动生成[J]. 软件学报, 1996, (6): 331- 338
PAN Zhi-geng, MA Xiao-hu, ZHANG Ming-min, et al The fourier descriptor based automatic generation method for multiple Chinese fonts[J]. Journal of Software, 1996, (6): 331- 338
[33]   杨丰, 余英林, 冯久超 基于B-样条小波变换汉字字形生成方法的研究[J]. 计算机工程与应用, 1998, (8): 23- 25
YANG Feng, YU Ying-lin, FENG Jiu-chao Generating new Chinese font with B-spline wavelet transform[J]. Computer Engineering and Applications, 1998, (8): 23- 25
doi: 10.3321/j.issn:1002-8331.1998.08.010
[34]   杨建, 张明敏, 张纪文, 等 基于C-Bézier曲线的汉字轮廓字库描述及生成[J]. 计算机辅助设计与图形学学报, 2000, (9): 660- 663
YANG Jian, ZHANG Ming-min, ZHANG Ji-wen, et al Description and generation of Chinese outline based on C-Bézier curves[J]. Journal of Computer-Aided Design and Computer Graphics, 2000, (9): 660- 663
doi: 10.3321/j.issn:1003-9775.2000.09.004
[35]   ZONG A, ZHU Y. StrokeBank: automating personalized Chinese handwriting generation [C]// Proceedings of the 28th AAAI Conference on Artificial Intelligence. Québec: AAAI, 2014, 4: 3024-3029.
[36]   栗青生, 徐强, 肖建国, 等 汉字动态生成的结构与风格模型[J]. 北京大学学报:自然科学版, 2017, 53 (2): 219- 229
LI Qing-sheng, XU Qiang, XIAO Jian-guo, et al A structure and style model for Chinese character dynamic generation[J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2017, 53 (2): 219- 229
[37]   LI Q S, YANG Y X A human-computer interactive dynamic description method for Jiaguwen characters[J]. Procedia Engineering, 2012, 29: 1013- 1017
doi: 10.1016/j.proeng.2012.01.081
[38]   栗青生, 熊晶, 吴琴霞, 等 基于特征加权的汉字点笔画生成研究[J]. 北京大学学报:自然科学版, 2014, 50 (1): 153- 160
LI Qing-sheng, XIONG Jing, WU Qin-xia, et al Study of feature weighted-based generation method for dian strokes of Chinese character[J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2014, 50 (1): 153- 160
[39]   LI Q S, LI X. Dynamic generation and editing system for wrongly written Chinese characters font [J]. Mathematical Problems in Engineering, 2015: 282408.
[40]   LIAN Z, XIAO J. Automatic shape morphing for chinese characters [C]// SIGGRAPH Asia 2012 Technical Briefs. New York: ACM, 2012: 1-4.
[41]   LIN J W, WANG C Y, TING C L, et al. Font generation of personal handwritten Chinese characters [C]// 5th International Conference on Graphic and Image Processing. Hong Kong: SPIE, 2013(9069): 1T.
[42]   CAMPBELL N D F, KAUTZ J Learning a manifold of fonts[J]. ACM Transactions on Graphics, 2014, 33 (4): 1- 11
[43]   PHAN H, FU H, CHAN A FlexyFont: learning transferring rules for flexible typeface synthesis[J]. Computer Graphics Forum, 2015, 34 (7): 245- 256
doi: 10.1111/cgf.12763
[44]   LIAN Z H, ZHAO B, XIAO J G. Automatic generation of large-scale handwriting fonts via style learning [C]//SIGGRAPH ASIA 2016 Technical Briefs. New York: ACM, 2016, 12: 1-4.
[45]   BERNHARDSSON E. Analyzing 50k fonts using deep neural networks [EB/OL]. [2021-05-26]. https://erikbern.com/2016/01/21analyzing-50k-fonts-using-deep-neural-networks.
[46]   BALUJA S Learning typographic style: from discrimination to synthesis[J]. Machine Vision and Applications, 2017, 28 (5/6): 551- 568
[47]   GUO Y, LIAN Z H, TANG Y M, et al. Creating new Chinese fonts based on manifold learning and adversarial networks [C]// Eurographics 2018. Delft: Eurographics Association, 2018: 61–64.
[48]   JIANG Y, LIAN Z H, TANG Y M Y, et al. SCFont: structure-guided Chinese font generation via deep stacked networks [C]// Proceedings of the AAAI Conference on Artificial Intelligence. Québec: AAAI, 2019, 33: 4015-4022.
[49]   ZHANG X, YIN F, ZHANG Y, et al Drawing and recognizing Chinese characters with recurrent neural network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40 (4): 849- 862
doi: 10.1109/TPAMI.2017.2695539
[50]   CHANG B, ZHANG Q, PAN S, et al. Generating handwritten Chinese characters using CycleGAN [C]// 2018 IEEE Winter Conference on Applications of Computer Vision. Lake Tahoe: IEEE, 2018: 199-207.
[51]   CHANG J, GU Y, ZHANG Y, et al. Chinese handwriting imitation with hierarchical generative adversarial network [C]// 29th British Machine Vision Conference. Newcastle: British Machine Vision Association, 2018: 290.
[52]   TIAN Y. Zi2zi [CP/OL]. [2021-05-26]. https://github.com/kaonashi-tyc/zi2zi.
[53]   XIAO F, ZHANG J, HUANG B, et al. Multiform fonts-to-fonts translation via style and content disentangled representations of Chinese character [EB/OL]. [2021-07-06]. https://arxiv.org/abs/2004.03338.
[54]   ZHANG Y X, ZHANG Y, CAI W B. Separating style and content for generalized style transfer [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE, 2018: 8447-8455.
[55]   ZHU A, LU X, BAI X, et al Few-shot text style transfer via deep feature similarity[J]. IEEE Transactions on Image Processing, 2020, 29: 6932- 6946
doi: 10.1109/TIP.2020.2995062
[56]   JIANG H C, YANG G Y, HUANG K Z, et al. W-Net: one-shot arbitrary-style Chinese character generation with deep neural networks [C]// International Conference on Neural Information Processing. Berlin: Springer, 2018: 483-493.
[57]   WU S J, YANG C Y, HSU J. CalliGAN: style and structure-aware Chinese calligraphy character generator [EB/OL]. (2020-05-26)[2021-07-06]. https://arxiv.org/abs/2005.12500.
[58]   WANG Y Z, GAO Y, LIAN Z H Attribute2Font: creating fonts you want from attributes[J]. ACM Transactions on Graphics, 2020, 39: 1- 15
[59]   HUANG G, LIU Z, MAATEN V D L, et al. Densely connected convolutional networks [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE, 2017: 2261-2269.
[60]   ZHU J Y, TAESUNG P, PHILLIP I, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks [C]// 2017 IEEE International Conference on Computer Vision. Los Alamitos: IEEE, 2017: 2242-2251.
[61]   TENENBAUM J B, FREEMAN W T Separating style and content with bilinear models[J]. Neural Computation, 2000, 12 (6): 1247- 1283
doi: 10.1162/089976600300015349
[62]   ANDERSON D O An introduction to bilinear time series models[J]. Journal of the Royal Statistical Society, 1979, 28 (3): 305- 306
[63]   ISOLA P, ZHU J, ZHOU T, et al. Image-to-image translation with conditional adversarial networks [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE, 2017: 5967-5976.
[64]   JOHNSON M, SCHUSTER M, LE Q V, et al Google's multilingual neural machine translation system: enabling zero-shot translation[J]. Transactions of the Association for Computational Linguistics, 2017, 5 (2): 339- 351
[65]   ODENA A, OLAH C, SHLENS J. Conditional image synthesis with auxiliary classifier GANs [C]// Proceedings of the 34th International Conference on Machine Learning. Sydney: PMLR, 2017, 70: 2642-2651.
[66]   YANIV T, ADAM P, LIOR W. Unsupervised cross-domain image generation [EB/OL]. [2021-07-06]. https://arxiv.org/abs/1611.02200.
[67]   ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein generative adversarial networks [C]// Proceedings of the 34th International Conference on Machine Learning. Sydney: PMLR, 2017, 70: 214-223.
[68]   HOCHREITER S, SCHMIDHUBER J Long short-term memory[J]. Neural Computation, 1997, 9 (8): 1735- 1780
doi: 10.1162/neco.1997.9.8.1735
[69]   LIU C L, YIN F, WANG D H, et al. CASIA online and offline Chinese handwriting database [DB/OL]. [2021-05-26]. http://www.nlpr.ia.ac.cn/databases/handwriting/home.html.
[70]   PAN W Q, LIAN Z H, SUN R J, et al. FlexiFont: a flexible system to generate personal font libraries [C]// Proceedings of the 2014 ACM Symposium on Document Engineering. New York: ACM, 2014: 17-20.
[71]   赵忠. 中国书法文字图像库[DB/OL]. [2021-05-26]. http://163.20.160.14/˜word/modules/myalbum/
[72]   WANG Z, BOVIK A C, SHEIKH H R, et al Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13 (4): 600- 612
[73]   SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 2818–2826.
[1] Jing-hui CHU,Li-dong SHI,Pei-guang JING,Wei LV. Context-aware knowledge distillation network for object detection[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(3): 503-509.
[2] Ting WANG,Xiao-fei ZHU,Gu TANG. Knowledge-enhanced graph convolutional neural networks for text classification[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 322-328.
[3] Pei-zhi WEN,Jun-mou CHEN,Yan-nan XIAO,Ya-yuan WEN,Wen-ming HUANG. Underwater image enhancement algorithm based on GAN and multi-level wavelet CNN[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 213-224.
[4] Guang-long LI,De-rong SHEN,Tie-zheng NIE,Yue KOU. Learning query optimization method based on multi model outside database[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 288-296.
[5] Tong CHEN,Jian-feng GUO,Xin-zhong HAN,Xue-li XIE,Jian-xiang XI. Visible and infrared image matching method based on generative adversarial model[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(1): 63-74.
[6] Song REN,Qian-wen ZHU,Xin-yue TU,Chao DENG,Xiao-shu WANG. Lining disease identification of highway tunnel based on deep learning[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(1): 92-99.
[7] Fa-ming HUANG,Li-han PAN,Chi YAO,Chuang-bing ZHOU,Qing-hui JIANG,Zhi-lu CHANG. Landslide susceptibility prediction modelling based on semi-supervised machine learning[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(9): 1705-1713.
[8] Xing LIU,Jian-bo YU. Attention convolutional GRU-based autoencoder and its application in industrial process monitoring[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(9): 1643-1651.
[9] Xue-yun CHEN,Xiao-qiao HUANG,Li XIE. Classification and detection method of blood cells images based on multi-scale conditional generative adversarial network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(9): 1772-1781.
[10] Nan ZHANG,Hong-zhao DONG,Yi-ni SHE. Seq2Seq prediction of bus trajectory on exclusive bus lanes[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(8): 1482-1489.
[11] Jia-cheng LIU,Jun-zhong JI. Classification method of fMRI data based on broad learning system[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(7): 1270-1278.
[12] Li-sheng JIN,Qiang HUA,Bai-cang GUO,Xian-yi XIE,Fu-gang YAN,Bo-tao WU. Multi-target tracking of vehicles based on optimized DeepSort[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(6): 1056-1064.
[13] Fei WANG,Wei-xiang XU. Improved model of road impedance function based on LSTM neural network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(6): 1065-1071.
[14] Yan HAO,Ya-bin DING,Jin-sheng FU. Hierarchical closed-loop optimization strategy for cumulative error of robot machining system[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(6): 1142-1149.
[15] Jia-hui XU,Jing-chang WANG,Ling CHEN,Yong WU. Surface water quality prediction model based on graph neural network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(4): 601-607.