1. Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China 2. Yunnan Key Laboratory of Computer Technologies Application, Kunming University of Science and Technology, Kunming 650500, China
The multimodal cascaded document layout analysis network (MCOD-Net) based on Transformer was proposed in order to solve the issue of misalignment in the existing methods for pretraining objectives in both text and image modalities, which involve complex preprocessing of document images using convolutional neural network (CNN) structures leading to many model parameters. The word block alignment embedding module (WAEM) was introduced to achieve alignment embedding of the pretraining objectives for text and image modalities. Masked language modeling (MLM), masked image modeling (MIM) and word-patch alignment (WPA) were utilized for pretraining in order to enhance the model’s representation learning capabilities across text and image modalities. The model structure was simplified and the number of model parameters was reduced by directly using the original document images and representing them using linear projected features of image blocks. The experimental results demonstrate that the proposed model achieves an mean average precision (mAP) of 95.1% on the publicly available PubLayNet dataset. A 2.5% overall performance improvement was achieved with outstanding generalization ability and exhibiting the best comprehensive performance compared with other models.
Tab.1Comparison of parameter sizes for different image processing methods
Fig.2PubLayNet dataset and corresponding labels
模型
主干网络
mAP/%
PublayNet[24]
Mask R-CNN
91.0
DiT[13]
Mask R-CNN
91.6
DiT[13]
Cascader R-CNN
92.5
UDoc[25]
Faster R-CNN
91.7
BEiT[23]
Mask R-CNN
92.6
MCOD-Net
Cascader R-CNN
95.1
Tab.2Overall performance of proposed model and existing models on PublayNet dataset
模型
mAP/%
文本
标题
列表
表格
图片
PublayNet[24]
91.6
84.0
88.6
96.0
94.9
DiT[13](Mask R-CNN)
92.8
84.5
86.8
97.5
96.5
DiT[13](Cascader R-CNN)
93.6
85.9
89.7
97.6
96.9
UDoc[25]
92.6
86.5
92.4
96.5
95.4
BEiT[23]
92.5
86.2
93.1
97.3
95.7
MCOD-Net
94.4
90.5
95.4
97.8
97.0
Tab.3mAP values of proposed model and existing models for identification of various elements in PublayNet dataset
Fig.3Recognition results of various elements
Fig.4Visualization of MCON-Net model on PublayNet dataset
Fig.5Visualization of MDOC-Net model on aluminum-silicon alloy literature
[1]
SOTO C, YOO S. Visual detection with context for document layout analysis [C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Hong Kong: ACL, 2019: 3464-3470.
[2]
WATANABE T, LUO Q, SUGIE N Structure recognition methods for various types of documents[J]. Machine Vision and Applications, 1993, 6 (2/3): 163- 176
[3]
HIRAYAMA Y. A method for table structure analysis using DP matching[C]//Proceedings of 3rd International Conference on Document Analysis and Recognition. Montreal: IEEE, 1995: 583-586.
[4]
FANG J, GAO L, BAI K, et al. A table detection method for multipage pdf documents via visual seperators and tabular structures [C]//2011 International Conference on Document Analysis and Recognition. Beijing: IEEE, 2011: 779-783.
[5]
BUNKE H, RIESEN K Recent advances in graph-based pattern recognition with applications in document analysis[J]. Pattern Recognition, 2011, 44 (5): 1057- 1067
doi: 10.1016/j.patcog.2010.11.015
[6]
HINTON G E, SALAKHUTDINOV R R Reducing the dimensionality of data with neural networks[J]. Science, 2006, 313 (5786): 504- 507
doi: 10.1126/science.1127647
SAHA R, MONDAL A, JAWAHAR C V. Graphical object detection in document images [C]//International Conference on Document Analysis and Recognition. Sydney: IEEE, 2019: 51-58.
[9]
GIRSHICK R. Fast r-cnn [C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 1440-1448.
[10]
HE K, GKIOXARI G, DOLLÁR P, et al. Mask r-cnn [C]//Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2961-2969.
[11]
RIBA P, DUTTA A, GOLDMANN L, et al. Table detection in invoice documents by graph neural networks [C]//2019 International Conference on Document Analysis and Recognition. Sydney: IEEE, 2019: 122-127.
[12]
XU Y, LI M, CUI L, et al. Layoutlm: pre-training of text and layout for document image understanding [C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. San Diego: ACM, 2020: 1192-1200.
[13]
LI J, XU Y, LV T, et al. Dit: self-supervised pre-training for document image transformer[C]//Proceedings of the 30th ACM International Conference on Multimedia. Lisson: ACM, 2022: 3530-3539.
[14]
APPALARAJU S, JASANI B, KOTA B U, et al. Docformer: end-to-end transformer for document understanding [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 993-1003.
[15]
PHAM V, PHAM C, DANG T. Road damage detection and classification with detectron2 and faster r-cnn [C]//2020 IEEE International Conference on Big Data. Atlanta: IEEE, 2020: 5592-5601.
[16]
CAI Z, VASCONCELOS N. Cascade r-cnn: delving into high quality object detection [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6154-6162.
[17]
RANFTL R, BOCHKOVSKIY A, KOLTUN V. Vision transformers for dense prediction [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 12179-12188.
[18]
KIM W, SON B, KIM I. Vilt: vision-and-language transformer without convolution or region supervision [C]// Proceedings of the 38th International Conference on Machine Learning. [S. l.]: PMLR, 2021: 5583-5594.
[19]
GHIASI G, LIN T Y, LE Q V. Nas-FPN: learning scalable feature pyramid architecture for object detection [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE , 2019: 7036-7045.
[20]
KAWINTIRANON K, SINGH L. Knowledge enhanced masked language model for stance detection [C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Mexico: ACL, 2021: 4725-4735.
[21]
XIE Z, ZHANG Z, CAO Y, et al. Simmim: a simple framework for masked image modeling [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 9653-9663.
[22]
HUANG Y, LV T, CUI L, et al. Layoutlmv3: pre-training for document ai with unified text and image masking [C]//Proceedings of the 30th ACM International Conference on Multimedia. Lisson: ACM, 2022: 4083-4091.
[23]
BAO H, DONG L, PIAO S, et al. Beit: Bert pre-training of image transformers [EB/OL]. [2022-09-03]. https://arxiv.org/abs/2106.08254.
[24]
ZHONG X, TANG J, YEPES A J. Publaynet: largest dataset ever for document layout analysis [C]//2019 International Conference on Document Analysis and Recognition. Sydney: IEEE, 2019: 1015-1022.
