|
|
Visible and infrared image matching method based on generative adversarial model |
Tong CHEN1(),Jian-feng GUO1,Xin-zhong HAN2,Xue-li XIE1,Jian-xiang XI1,*() |
1. Department of Missile Engineering, Rocket Force University of Engineering, Xi’an 710000, China 2. The 96901 Unit of the Chinese People’s Liberation Army, Beijing 100094, China |
|
|
Abstract A visible-infrared image matching method based on generative adversarial model was proposed based on the style transfer of generative adversarial network and traditional local feature extraction capability in order to analyze the problems of large modal difference, difficult matching and poor robustness of existing multi-sensor images matching methods. The loss function calculation path was added and a new loss function was constructed according to the idea of style transfer in GAN network in order to improve the transfer effect of the model on the multi-sensor images. The feature information of the transformed homologous images was extracted by using SIFT algorithm. Then the position and scale of the points to be matched were determined. The feature matching and similarity measurement between the two images were indirectly completed according to the matching strategy. Experiments were conducted on the realistic aerial dataset. Results show that the proposed method can effectively deal with multi-modal data and reduce the difficulty of multi-sensor image matching. The method can provide a new solution for multi-sensor images matching.
|
Received: 19 June 2021
Published: 05 January 2022
|
|
Fund: 国家自然科学基金资助项目(62176263, 62103434); 陕西省杰出青年科学基金资助项目(2021JC-35); 中国博士后科学基金特别资助项目(2021T140790) |
Corresponding Authors:
Jian-xiang XI
E-mail: chentong00817@163.com;xijx07@mails.tsinghua.edu.cn
|
基于生成对抗模型的可见光-红外图像匹配方法
针对现有异源图像匹配存在的模态差异大、匹配难度大、鲁棒性差等问题, 基于生成对抗网络转换思想及传统的局部特征提取能力, 提出基于生成对抗模型的可见光-红外图像匹配方法. 依据生成对抗网络(GAN)的风格转换思想, 增加了损失函数计算通路并构建新的损失函数, 改进模型在异源图像上的转换效果. 利用SIFT算法分别提取转换后同源图像的特征信息, 确定待匹配点的位置和尺度. 依据匹配策略间接完成待配准图像的特征匹配及相似性度量. 在实景航拍数据集上进行实验验证. 结果表明, 利用该方法能够有效地处理多模数据, 降低异源图像的匹配难度, 为多模态图像匹配问题提供新的思路.
关键词:
航拍图像处理,
异源图像匹配,
深度学习,
生成对抗网络(GAN),
风格转换
|
|
[1] |
BROWN L G A survey of image registration techniques[J]. ACM Computer Surveys (CSUR), 1992, 24 (4): 325- 376
doi: 10.1145/146370.146374
|
|
|
[2] |
BARBARA Z, FLUSSER J Image registration methods: a survey[J]. Image and Vision Computing, 2003, 21 (11): 977- 1000
doi: 10.1016/S0262-8856(03)00137-9
|
|
|
[3] |
XIONG Z, ZHANG Y A critical review of image registration methods[J]. International Journal of Image and Data Fusion, 2010, 1 (2): 137- 158
doi: 10.1080/19479831003802790
|
|
|
[4] |
SHI W, SU F, WANG R, et al. A visual circle based image registration algorithm for optical and SAR imagery [C]// IEEE International Geoscience and Remote Sensing Symposium. Munich: IEEE, 2012: 2109-2112.
|
|
|
[5] |
SURI S, REINARTZ P Mutual-information-based registration of terraSAR-X and ikonos imagery in urban areas[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 48 (2): 939- 949
doi: 10.1109/TGRS.2009.2034842
|
|
|
[6] |
HASAN M, PICKERING M R, JIA X Robust automatic registration of multimodal satellite images using CCRE with partial volume interpolation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2012, 50 (10): 4050- 4061
doi: 10.1109/TGRS.2012.2187456
|
|
|
[7] |
高雪艳, 潘安宁, 杨扬 基于图像混合特征的城市绿地遥感图像配准[J]. 浙江大学学报: 工学版, 2019, 53 (6): 1205- 1217 GAO Xue-yan, PAN An-ning, YANG Yang Urban green space remote sensing image registration using image mixed features[J]. Journal of Zhejiang University: Engineering Science, 2019, 53 (6): 1205- 1217
doi: 10.3785/j.issn.1008-973X.2019.06.021
|
|
|
[8] |
张莉, 刘济林 多摄像机间基于最稳定极值区域的人体跟踪方法[J]. 浙江大学学报: 工学版, 2010, 44 (6): 1091- 1097 ZHANG Li, LIU Ji-Lin Human tracking method based on maximally stable extremal regions with multi cameras[J]. Journal of Zhejiang University: Engineering Science, 2010, 44 (6): 1091- 1097
doi: 10.3785/j.issn.1008-973X.2010.06.007
|
|
|
[9] |
REN S Q, HE K M, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149
doi: 10.1109/TPAMI.2016.2577031
|
|
|
[10] |
LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection [C]// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2117-2125.
|
|
|
[11] |
胡惠雅, 盖绍彦, 达飞鹏 基于生成对抗网络的偏转人脸转正[J]. 浙江大学学报: 工学版, 2021, 55 (1): 116- 123 HU Hui-ya, GAI Shao-yan, DA Fei-peng Face frontalization based on generative adversarial network[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (1): 116- 123
|
|
|
[12] |
GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al Generative adversarial networks[J]. Communications of the ACM, 2020, 63 (11): 139- 144
doi: 10.1145/3422622
|
|
|
[13] |
DENTON E L, CHINTALA S, SZLAM A, et al Deep generative image models using a laplacian pyramid of adversarial networks[J]. Computer Vision and Pattern Recognition, 2015, 6 (1): 1486- 1494
|
|
|
[14] |
ZHU J Y, PHILIPP K, SHECHTMAN E, et al. Generative visual manipulation on the natural image manifold [C]// European Conference on Computer Vision. Cham: Springer, 2016: 597-613.
|
|
|
[15] |
LI C, WAND M. Precomputed real-time texture synthesis with markovian generative adversarial networks [C]// European Conference on Computer Vision. Cham: Springer, 2016: 702-716.
|
|
|
[16] |
唐贤伦, 杜一铭, 刘雨微, 等 基于条件深度卷积生成对抗网络的图像识别方法[J]. 自动化学报, 2018, 44 (5): 855- 864 TANG Xian-lun, DU Yi-ming, LIU Yu-wei, et al Image recognition with conditional deep convolutional generative adversarial networks[J]. Acta Automatica Sinica, 2018, 44 (5): 855- 864
|
|
|
[17] |
RADFORD A, METZ L, CHINTALA S, et al. Unsupervised representation learning with deep convolutional generative adversarial networks [C]// International Conference of Legal Regulators. Washington: [s. n.], 2016: 1-16.
|
|
|
[18] |
ARJOVSKY M, CHINTALA S, BOTTOU L, et al. Wasserstein generative adversarial networks [C]// International Conference on Machine Learning. Sydney: [s. n. ], 2017: 214-223.
|
|
|
[19] |
ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks [C]// IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2223-2232.
|
|
|
[20] |
HUANG X, LIU M Y, BELONGIE S, et al. Multimodal unsupervised image-to-image translation [C]// European Conference on Computer Vision. Cham: Springer, 2018: 172-189.
|
|
|
[21] |
LEE H Y, TSENG H Y, HUANG J B, et al. Diverse image-to-image translation via disentangled representations [C]// European Conference on Computer Vision. Cham: Springer, 2018: 35-51.
|
|
|
[22] |
CHANG H Y, WANG Z, CHUANG Y Y. Domain-specific mappings for generative adversarial style transfer [C]// European Conference on Computer Vision. Cham: Springer, 2020: 573-589.
|
|
|
[23] |
SONG L, ZHANG M, WU X, et al. Adversarial discriminative heterogeneous face recognition [C]// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. New Orleans: AAAI, 2018: 7355-7362.
|
|
|
[24] |
MERKLE N, AUER S, MULLER R, et al Exploring the potential of conditional adversarial networks for optical and SAR image matching[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11 (6): 1811- 1820
doi: 10.1109/JSTARS.2018.2803212
|
|
|
[25] |
胡麟苗, 张湧 基于生成对抗网络的短波红外-可见光人脸图像翻译[J]. 光学学报, 2020, 40 (5): 0510001 HU Lin-miao, ZHANG Yong Facial image translation in short-wavelength infrared and visible light based on generative adversarial network[J]. Acta Optica Sinica, 2020, 40 (5): 0510001
doi: 10.3788/AOS202040.0510001
|
|
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
|
Shared |
|
|
|
|
|
Discussed |
|
|
|
|