Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2026, Vol. 60 Issue (12): 2694-2703    DOI: 10.3785/j.issn.1008-973X.2026.12.015
    
Image semantic communication based on CSI-guided attention and gated generation repair
Yi PENG(),Heng LI,Qingqing YANG*(),Hui LI,Jianming WANG
Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650031, China
Download: HTML     PDF(1392KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A Swin Transformer-based image semantic communication scheme that integrated channel state information (CSI) awareness with generative adversarial network (GAN) assistance was proposed aiming at the issues of texture loss and edge blurring in reconstructed images encountered by existing deep joint source-channel coding schemes for image semantic communication in order to improve the perceptual quality and robustness of image transmission under typical SNR conditions. A CSI-guided self-attention mechanism was designed at the encoder. A scalar CSI was mapped into an additive bias matrix through a multilayer perceptron and incorporated into the self-attention computation in order to dynamically reshape the feature extraction topology. A CSI-gated generative restoration module was developed at the decoder. CSI was nonlinearly mapped to generate a gating factor for dynamically adjusting the restoration weight of the GAN on residual features. The experimental results demonstrate that the proposed scheme achieves superior peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) performance across datasets with different resolutions compared with conventional separated source-channel coding and existing mainstream semantic communication schemes. The proposed method provides an effective solution for image semantic transmission in complex channel environments.



Key words: semantic communication      Swin Transformer      channel state information      adaptive adjustment      generative adversarial network     
Received: 04 December 2025      Published: 21 October 2026
CLC:  TN 919  
  TP 183  
  TP 391  
Fund:  国家自然科学基金资助项目(62461030);云南省基础研究重点资助项目(202401AS070105);中央引导地方科技发展专项资金资助项目(202607AD040003).
Corresponding Authors: Qingqing YANG     E-mail: 12309214@kust.edu.cn;20090119@kust.edu.cn
Cite this article:

Yi PENG,Heng LI,Qingqing YANG,Hui LI,Jianming WANG. Image semantic communication based on CSI-guided attention and gated generation repair. Journal of ZheJiang University (Engineering Science), 2026, 60(12): 2694-2703.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2026.12.015     OR     https://www.zjujournals.com/eng/Y2026/V60/I12/2694


基于CSI导向注意力与门控生成修复的图像语义通信

针对现有深度联合信源信道编码方案在图像语义通信中存在重建纹理丢失与边缘模糊的问题,提出融合信道状态信息(CSI)感知与生成对抗网络(GAN)辅助的Swin Transformer图像语义通信方案,以提升典型信噪比下图像传输的感知质量与鲁棒性. 在编码端,设计CSI导向自注意力机制,通过多层感知机将标量CSI映射为加性偏置矩阵并注入到自注意力计算中,动态重塑特征提取拓扑. 在解码端,构建CSI门控生成式修复模块,利用CSI经过非线性映射生成门控因子,自适应调节GAN对残差特征的修复权重. 实验表明,所提方案在不同分辨率数据集上的峰值信噪比(PSNR)与多尺度结构相似性(MS-SSIM)指标均优于传统分离式编码及现有主流语义通信方案,为复杂信道环境下的图像语义传输提供了有效的解决方案.


关键词: 语义通信,  Swin Transformer,  信道状态信息,  自适应调节,  生成对抗网络 
Fig.1 Image semantic transmission system based on Swin Transformer
Fig.2 Two consecutive Swin Transformer blocks
Fig.3 CSI perception dual-end adaptive mechanism
Fig.4 Structure of CGSA mechanism
Fig.5 Structure of CGGR module
Fig.6 PSNR curves of reconstructed images by different transmission schemes varying with signal-to-noise ratio
Fig.7 MS-SSIM curves of reconstructed images by different transmission schemes varying with signal-to-noise ratio
Fig.8 Visual comparison of typical scenarios for transmitting Kodak dataset using different schemes
方案变体描述
CAG-ST本研究提出的完整方案,同时包含CGSA机制和CGGR模块.
CAG-ST(CGSA)在基线基础上,仅在编码端引入CSI导向自注意力机制.
CAG-ST(CGGR,$ \alpha $+GAN)在基线基础上,仅在解码端引入完整的CGGR模块.
CAG-ST(CGGR,$ \alpha $)CGGR模块移除GAN仅保留CSI门控机制,且编码端不含CGSA机制.
CAG-ST(CGGR,GAN)CGGR模块移除CSI门控机制仅保留GAN,且编码端不含CGSA机制.
Tab.1 Different variants of CAG-ST scheme
Fig.9 PSNR curves of reconstructed images for different variants of CAG-ST varying with signal-to-noise ratio
Fig.10 MS-SSIM curves of reconstructed images for different variants of CAG-ST varying with signal-to-noise ratio
传输方案Np/106FLOPs/109ten/mstde/ms
Deep JSCC15.8153.4348.2667.48
WITT28.233.617.519.39
CAG-ST(CGSA)37.5341.379.199.72
CAG-ST45.4775.989.1915.41
Tab.2 Complexity testing of different transmission schemes
[1]   Tong W, Zhu P. 6G: the next horizon [M]. Cambridge: Cambridge University Press, 2021.
[2]   Letaief K B, Chen Wei, Shi Yuanming, et al The roadmap to 6G: AI empowered wireless networks[J]. IEEE Communications Magazine, 2019, 57 (8): 84- 90
doi: 10.1109/MCOM.2019.1900271
[3]   Wang Y, Han H, Feng Y, et al Semantic communication empowered 6g networks: techniques, applications, and challenges[J]. IEEE Access, 2025, 13: 28293- 28314
doi: 10.1109/ACCESS.2025.3532797
[4]   栾宏之, 李书才, 李屹, 等 5G-A车联网确定性网络中的语义通信系统[J]. 浙江大学学报(工学版), 2025, 59 (5): 964- 972
Luan Hongzhi, Li Shucai, Li Yi, et al Semantic communication system in deterministic networking for5G-advanced vehicular network[J]. Journal of Zhejiang University (Engineering Science), 2025, 59 (5): 964- 972
[5]   Shannon C E A mathematical theory of communication[J]. The Bell System Technical Journal, 1948, 27 (3): 379- 423
doi: 10.1002/j.1538-7305.1948.tb01338.x
[6]   Zhang P, Xu W, Gao H, et al Toward wisdom-evolutionary and primitive-concise 6G: a new paradigm of semantic communication networks[J]. Engineering, 2022, 8: 60- 73
doi: 10.1016/j.eng.2021.11.003
[7]   Fang Y, He Q, Ding D, et al. Deep-learning based gaussian mixture likelihoods joint source-channel coding for image transmission [C]//2023 International Conference on Wireless Communications and Signal Processing. Hangzhou: IEEE, 2023: 19-24.
[8]   Gündüz D, Qin Z, Aguerri I E, et al Beyond transmitting bits: context, semantics, and task-oriented communications[J]. IEEE Journal on Selected Areas in Communications, 2023, 41 (1): 5- 41
doi: 10.1109/JSAC.2022.3223408
[9]   Bourtsoulatze E, Kurka D B, Gündüz D Deep joint source-channel coding for wireless image transmission[J]. IEEE Transactions on Cognitive Communications and Networking, 2019, 5 (3): 567- 579
doi: 10.1109/TCCN.2019.2919300
[10]   Dai J, Wang S, Tan K, et al Nonlinear transform source-channel coding for semantic communications[J]. IEEE Journal on Selected Areas in Communications, 2022, 40 (8): 2300- 2316
doi: 10.1109/JSAC.2022.3180802
[11]   Park T, Hong E, Jeon Y, et al. Robust deep joint source channel coding for task-oriented semantic communications [PP/OL]. arXiv (2025-03-17)[2025-11-15]. https://arxiv.org/abs/2503.12907.
[12]   Hu Q, Zhang G, Qin Z, et al Robust semantic communications with masked VQ-VAE enabled codebook[J]. IEEE Transactions on Communications, 2023, 22 (12): 8707- 8722
[13]   Kurka D B, Gündüz D DeepJSCC-f: deep joint source-channel coding of images with feedback[J]. IEEE Journal on Selected Areas in Information Theory, 2020, 1 (1): 178- 193
doi: 10.1109/JSAIT.2020.2987203
[14]   Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16×16 words: Transformers for image recognition at scale [PP/OL]. arXiv (2020-10-22)[2025-11-15]. https://arxiv.org/abs/2010.11929.
[15]   Liu Z, Lin Y, Cao Y, et al. Swin transformer: hierarchical vision transformer using shifted windows [C]// IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 10012-10022.
[16]   张振利, 胡新凯, 李凡, 等 基于 CNN 和 Efficient Transformer 的多尺度遥感图像语义分割算法[J]. 浙江大学学报(工学版), 2025, 59 (4): 778- 786
Zhang Zhenli, Hu Xinkai, Li Fan, et al Semantic segmentation algorithm for multiscale remote sensing images based on CNN and Efficient Transformer[J]. Journal of Zhejiang University (Engineering Science), 2025, 59 (4): 778- 786
[17]   刘春娟, 乔泽, 闫浩文, 等 基于多尺度互注意力的遥感图像语义分割网络[J]. 浙江大学学报(工学版), 2023, 57 (07): 1335- 1344
Liu Chunjuan, Qiao Ze, Yan Haowen, et al Semantic segmentation network for remote sensing image based on multi-scale mutual attention[J]. Journal of Zhejiang University (Engineering Science), 2023, 57 (07): 1335- 1344
[18]   梁龙学, 贺成龙, 吴小所, 等 全局信息提取与重建的遥感图像语义分割网络[J]. 浙江大学学报(工学版), 2024, 58 (11): 2270- 2279
Liang Longxue, He Chenglong, Wu Xiaosuo, et al Remote sensing image semantic segmentation network based on global information extraction and reconstruction[J]. Journal of Zhejiang University (Engineering Science), 2024, 58 (11): 2270- 2279
[19]   Yang K, Wang S, Dai J, et al. WITT: a wireless image transmission transformer for semantic communications [C]// 2023 IEEE International Conference on Acoustics, Speech and Signal Processing. Rhodes: IEEE, 2023: 1-5.
[20]   Nguyen L X, Tun Y L, Tun Y K, et al Swin transformer-based dynamic semantic communication for multi-user with different computing capacity[J]. IEEE Transactions on Vehicular Technology, 2024, 73 (6): 8957- 8972
doi: 10.1109/TVT.2024.3362328
[21]   Blau Y, Michaeli T. The perception-distortion tradeoff [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6228-6237.
[22]   Ledig C, Theis L, Huszár F, et al. Photo-realistic single image super-resolution using a generative adversarial network [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4681–4690.
[23]   Agustsson E, Minnen D, Toderici G, et al Generative adversarial networks for extreme learned image compression[J]. IEEE Transactions on Image Processing, 2020, 29: 2752- 2763
[24]   Zhang G, Hu Q, Cai Y, et al. Adaptive CSI feedback for deep learning-enabled image transmission [C]// IEEE International Conference on Communications. Rome: IEEE, 2023: 4163-4168.
[25]   Song M, Ma N, Dong C, et al Deep joint source-channel coding for wireless image transmission with adaptive models[J]. Electronics, 2023, 12 (22): 4637
doi: 10.3390/electronics12224637
[26]   Liang C, Du H, Sun Y, et al Generative AI-driven semantic communication networks: architecture, technologies and applications[J]. IEEE Transactions on Cognitive Communications and Networking, 2024, 11 (1): 27- 47
doi: 10.1109/tccn.2024.3435524
[27]   Grassucci E, Barbarossa S, Comminiello D. Generative semantic communication: diffusion models beyond bit recovery [PP/OL]. arXiv (2023-06-07)[2025-11-15]. https://arxiv.org/abs/2306.04321.
[28]   Erdem E, Gündüz D, Dragotti P L, et al Generative joint source-channel coding for semantic image transmission[J]. IEEE Journal on Selected Areas in Communications, 2023, 41 (8): 2645- 2657
doi: 10.1109/JSAC.2023.3288243
[29]   杨冰, 周家辉, 姚金良, 等 基于多模态语义信息的文本生成图像方法[J]. 浙江大学学报(工学版), 2026, 60 (2): 1- 10
Yang Bing, Zhou Jiahui, Yao Jinliang, et al Text-to-image generation method based on multimodal semantic information[J]. Journal of Zhejiang University (Engineering Science), 2026, 60 (2): 1- 10
[30]   Liang W, Ma N, Shen L, et al. An image semantic communication system based on Swin Transformer and generative adversarial networks [C]// 2024 IEEE Globecom Workshops. Cape Town: IEEE, 2024: 1-6.
[31]   Richardson T, Kudekar S Design of low-density parity check codes for 5G new radio[J]. IEEE Communications Magazine, 2018, 56 (3): 28- 34
doi: 10.1109/MCOM.2018.1700839
[1] Yunhong LI,Qiqi ZHANG,Jinni CHEN,Weichong CHEN,Xueping SU,Chengming LIANG. Text-to-image generation algorithm based on generative adversarial network and coordinate attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(6): 1213-1220.
[2] Kejia CHEN,Weiwen ZENG,Zhenqing WANG. Image data enhancement method based on conditional generative adversarial network and Poisson blending[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(12): 2592-2601.
[3] Li SONG,Yuanyu WAN,Canghong JIN,Mingli SONG. Dynamic Monte-Carlo reinforcement learning based on adversarial graph generation[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(12): 2613-2622.
[4] Yi PENG,Heng LI,Qingqing YANG,Hui LI,Jianming WANG. Multilevel semantic MIMO transmission of images based on Swin Transformer[J]. Journal of ZheJiang University (Engineering Science), 2026, 60(11): 2462-2471.
[5] Hongzhi LUAN,Shucai LI,Yi LI,Chuanjie QIAN,Qiong MEI. Semantic communication system in deterministic networking for5G-advanced vehicular network[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(5): 964-972.
[6] Yong LI,Zhiqiang LIU,Maoxing TIAN,Songlin JIA. Semantic compression algorithm based on global correlated semantic importance[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(4): 795-803.
[7] Yan YANG,Cunpeng JIA. An efficient image dehazing algorithm with Agent Attention for domain feature interaction[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(12): 2527-2538.
[8] Tianfei WANG,Wenjun ZHOU,Sheng XIANG,Yuhang HE,Bo PENG. Industrial image anomaly detection method based on adversarial learning of abnormal features[J]. Journal of ZheJiang University (Engineering Science), 2025, 59(12): 2566-2575.
[9] Lei YANG,Pengju HE,Xingxing CHOU. TimeGAN data augmentation-based fault classification method for complex processes[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(9): 1768-1780.
[10] Canlin LI,Xinyue WANG,Lizhuang MA,Zhiwen SHAO,Wenjiao ZHANG. Image cartoonization incorporating attention mechanism and structural line extraction[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(8): 1728-1737.
[11] Yin CAO,Junping QIN,Tong GAO,Qianli MA,Jiaqi REN. Generative adversarial network based two-stage generation of high-quality images from text[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(4): 674-683.
[12] Yin CAO,Junping QIN,Qianli MA,Hao SUN,Kai YAN,Lei WANG,Jiaqi REN. Survey of text-to-image synthesis[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(2): 219-238.
[13] Canlin LI,Wenjiao ZHANG,Zhiwen SHAO,Lizhuang MA,Xinyue WANG. Semantic segmentation method on nighttime road scene based on Trans-nightSeg[J]. Journal of ZheJiang University (Engineering Science), 2024, 58(2): 294-303.
[14] Hai-feng LI,Xue-ying ZHANG,Shu-fei DUAN,Hai-rong JIA,Hui-zhi LIANG. Fusing generative adversarial network and temporal convolutional network for Mandarin emotion recognition[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(9): 1865-1875.
[15] Yun-hong LI,Jiao-jiao DUAN,Xue-ping SU,Lei-tao ZHANG,Hui-kang YU,Xing-rui LIU. Calligraphy generation algorithm based on improved generative adversarial network[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(7): 1326-1334.