Please wait a minute...
浙江大学学报(工学版)  2023, Vol. 57 Issue (4): 744-752    DOI: 10.3785/j.issn.1008-973X.2023.04.012
自动化技术、计算机技术     
基于距离度量损失框架的半监督学习方法
刘半藤1,2(),叶赞挺2,秦海龙3,王柯1,4,*(),郑启航1,王章权1,2
1. 浙江树人学院 信息科技学院,浙江 杭州 310015
2. 常州大学 计算机与人工智能学院,江苏 常州 213164
3. 浙江绿城未来数智科技有限公司,浙江 杭州 311121
4. 浙江大学 工业控制技术国家重点实验室,浙江 杭州 310027
Semi-supervised learning method based on distance metric loss framework
Ban-teng LIU1,2(),Zan-ting YE2,Hai-long QIN3,Ke WANG1,4,*(),Qi-hang ZHENG1,Zhang-quan WANG1,2
1. College of Information Science and Technology, Zhejiang Shuren University, Hangzhou 310015, China
2. College of Computer Science and Artificial Intelligence, Changzhou University, Changzhou 213164, China
3. Zhejiang Lvcheng Future Digital Intelligence Technology Limited Company, Hangzhou 311121, China
4. State Key Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou 310027, China
 全文: PDF(1146 KB)   HTML
摘要:

为了解决半监督学习方法训练过程中因损失函数类型不同、损失尺度不统一而导致的损失权重难以调节, 模型优化方向不统一与泛化能力不足的问题, 提出基于距离度量损失框架的半监督学习方法. 该方法从距离度量损失的角度出发, 提出统一损失框架函数, 实现了半监督任务中不同损失函数之间的损失权重调节. 针对损失框架中嵌入向量的目标区域问题, 引入自适应相似度权重,以避免传统度量学习损失函数优化方向的冲突, 提高模型的泛化性能. 为了验证方法的有效性, 分别采用CNN13网络和ResNet18网络,在CIFAR-10、CIFAR-100、SVHN、STL-10标准图像数据集和医疗肺炎数据集Pneumonia Chest X-ray上,构建半监督学习模型与常用半监督方法进行比较. 实验结果表明, 在同等标签数目的条件下, 提出方法具有最优的分类准确度.

关键词: 半监督学习度量学习损失函数损失框架分类    
Abstract:

A semi-supervised learning method based on the distance metric loss framework was proposed in order to solve the problems of different types of loss functions and inconsistent loss scales in the training process of semi-supervised learning methods, which make it difficult to adjust the loss weights, inconsistent optimization directions and insufficient generalization ability. A unify loss framework function was proposed from the perspective of distance metric loss, and the adjustment of loss weights between different loss functions in semi-supervised tasks was achieved. Adaptive similarity weights were introduced for the target region problem of embedding vectors in the loss framework in order to avoid the conflict of optimization directions of traditional metric learning loss functions and improve the generalization performance of the model. CNN13 and ResNet18 networks were used to construct semi-supervised learning models on CIFAR-10, CIFAR-100, SVHN, STL-10 standard image dataset and medical pneumonia dataset Pneumonia Chest X-ray, respectively, for comparison with commonly used semi-supervised methods in order to validate the effectiveness of the method. Results show that the method has the optimal classification accuracy under the condition of the same number of labels.

Key words: semi-supervised learning    metric learning    loss function    loss framework    classification
收稿日期: 2022-04-07 出版日期: 2023-04-21
CLC:  TP 391  
基金资助: 浙江省“领雁”研发攻关计划资助项目(2022C03122);浙江省公益技术应用研究资助项目(LGF22F020006,LGF21F010004);浙江大学工业控制技术国家重点实验室开放课题资助项目(ICT2022B34)
通讯作者: 王柯     E-mail: hupo3@sina.com;wangke1992@zju.edu.cn
作者简介: 刘半藤(1984—),男,教授,从事复合无损技术的研究. orcid.org/0000-0001-8472-0061. E-mail: hupo3@sina.com
服务  
把本文推荐给朋友
加入引用管理器
E-mail Alert
作者相关文章  
刘半藤
叶赞挺
秦海龙
王柯
郑启航
王章权

引用本文:

刘半藤,叶赞挺,秦海龙,王柯,郑启航,王章权. 基于距离度量损失框架的半监督学习方法[J]. 浙江大学学报(工学版), 2023, 57(4): 744-752.

Ban-teng LIU,Zan-ting YE,Hai-long QIN,Ke WANG,Qi-hang ZHENG,Zhang-quan WANG. Semi-supervised learning method based on distance metric loss framework. Journal of ZheJiang University (Engineering Science), 2023, 57(4): 744-752.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2023.04.012        https://www.zjujournals.com/eng/CN/Y2023/V57/I4/744

图 1  一致性损失与熵最小化损失的示意图
图 2  嵌入向量与特征向量优化方法的示意图
图 3  Unify Loss函数曲面图
图 4  各损失梯度的对比图
数据集 N Nt
Nl Nu
SVHN 40 70 000 26 000
250 70 000
4 000 70 000
CIFAR-10 40 50 000 5 000
250 50 000
4 000 50 000
CIFAR-100 400 40 000 10 000
1 000 40 000
10 000 40 000
STL-10 1 000 10 000 8 000
5 000 10 000
Pneumonia Chest X-ray 250 4 750 600
500 4 500
1 000 4 000
表 1  各数据集的训练及测试样本分布
图 5  参数取值结果的对比图
方法 CIFAR-10 CIFAR-100 SVHN STL-10
Nl = 40 Nl = 250 Nl = 4000 Nl = 400 Nl = 1000 Nl = 10000 Nl = 40 Nl = 250 Nl = 4000 Nl = 1000 Nl = 5000
VAT 27.41 53.97 82.59 5.12 12.89 53.20 35.64 79.78 90.06 68.77 81.26
MeanTeacher 26.54 55.52 83.79 4.94 12.13 55.13 36.45 78.43 93.58 69.01 82.80
MixMatch 41.47 74.08 90.74 9.15 32.39 65.87 47.45 76.02 93.50 79.59 88.41
Remixmatch 42.32 78.71 92.20 12.98 44.56 72.33 52.55 81.92 94.67 84.33 92.89
UDA 44.55 81.66 92.45 10.66 40.72 71.18 47.37 84.31 95.54 82.34 92.74
UDA_unify 49.69 84.22 93.87 18.06 46.12 72.67 53.14 87.32 95.77 87.21 94.34
Fixmatch 50.17 86.28 93.11 12.65 42.95 72.08 56.41 88.75 95.94 89.68 94.58
Fixmatch_unify 55.24 89.94 94.74 18.51 47.79 73.38 65.86 90.09 96.02 93.44 95.25
Flexmatch 52.70 85.08 93.97 27.22 55.12 77.84 62.03 90.42 95.83 92.36 96.06
Unify Loss 61.19 90.33 94.25 35.35 61.75 78.11 67.35 92.47 96.17 93.42 96.51
表 2  各方法在CIFAR-10、CIFAR-100、SVHN、STL-10数据集中的准确率
方法 A/%
Nl/N = 5% Nl/N = 10% Nl/N = 20%
VAT 8.62 25.71 67.12
MeanTeacher 10.72 27.92 69.46
MixMatch 12.24 31.62 73.86
Remixmatch 15.07 33.78 74.02
UDA 11.87 31.89 75.31
UDA_unify 17.66 36.46 76.55
Fixmatch 19.12 37.22 77.54
Fixmatch_unify 23.55 40.91 77.67
Flexmatch 23.38 40.67 78.76
Unify Loss 26.57 42.26 79.24
表 3  各方法在 Pneumonia Chest X-ray 数据集中的验证结果
1 KORNBLITH S, SHLENS J, LE Q V. Do better imagenet models transfer better? [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 2661-2671.
2 YANG S, LUO P, LOY C C, et al. WIDER FACE: a face detection benchmark [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 5525-5533.
3 许佳辉, 王敬昌, 陈岭, 等 基于图神经网络的地表水水质预测模型[J]. 浙江大学学报:工学版, 2021, 55 (4): 601- 607
XU Jia-hui, WANG Jing-chang, CHEN Ling, et al Surface water quality prediction model based on graph neural network[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (4): 601- 607
4 LEE D H. Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks [C]// ICML 2013 Workshop on Challenges in Representation Learning. Atlanta: PMLR, 2013: 896.
5 TARVAINEN A, VALPOLA H Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results[J]. Advances in Neural Information Processing Systems, 2017, 30: 1195- 1204
6 XIE Q, DAI Z, HOVV E, et al Unsupervised data augmentation for consistency training[J]. Advances in Neural Information Processing Systems, 2020, 33 (2): 6256- 6268
7 MIYATO T, MAEDA S, KOYAMA M, et al Virtual adversarial training: a regularization method for supervised and semi-supervised learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41 (8): 1979- 1993
8 WANG F, CHENG J, LIU W, et al Additive margin softmax for face verification[J]. IEEE Signal Processing Letters, 2018, 25 (7): 926- 930
doi: 10.1109/LSP.2018.2822810
9 LAINE S, AILA T. Temporal ensembling for semi-supervised learning [C]// International Conference on Learning Representations. Toulon: [s. n.], 2017: 1-13.
10 SAJJADI M, JAVANMARDI M, TASDIZEN T Regularization with stochastic transformations and perturbations for deep semi-supervised learning[J]. Advances in Neural Information Processing Systems, 2016, 29 (7): 1163- 1171
11 LIU W, WEN Y, YU Z, et al. Large-margin softmax loss for convolutional neural networks [C]// Proceedings of the 33rd International Conference on Machine Learning. New York: PMLR, 2016: 507-516.
12 LI Y, GAO F, OU Z, et al. Angular softmax loss for end-to-end speaker verification [C]// 2018 11th International Symposium on Chinese Spoken Language Processing. Taipei: IEEE, 2018: 190-194.
13 GRANDVALET Y, BENGIO Y Semi-supervised learning by entropy minimization[J]. Advances in Neural Information Processing Systems, 2004, 17: 529- 536
14 VERMA V, KAWAGUCHI K, LAMB A, et al Interpolation consistency training for semi-supervised learning[J]. Neural Networks, 2022, 145: 90- 106
doi: 10.1016/j.neunet.2021.10.008
15 HENDRYCKS D, MU N, CUBUK E D, et al. Augmix: a simple method to improve robustness and uncertainty under data shift [C]// International Conference on Learning Representations. Ethiopia: [s. n.], 2020: 1-15.
16 KENDALL A, GAL Y, CIPOLLA R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7482-7491.
17 AZIERE N, TODOROVIC S. Ensemble deep manifold similarity learning using hard proxies [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 7299-7307.
18 QIAN Q, SHANG L, SUN B, et al. Softtriple loss: deep metric learning without triplet sampling [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 6450-6458.
19 KIM S, KIM D, CHO M, et al. Proxy anchor loss for deep metric learning [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 3238-3247.
20 SUN Y, CHENG C, ZHANG Y, et al. Circle loss: a unified perspective of pair similarity optimization [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 6398-6407.
21 KERMANY D, ZHANG K, GOLDBAUM M Labeled optical coherence tomography (oct) and chest X-ray images for classification[J]. Mendeley Data, 2018, 2 (2): 255- 265
22 BERTHELOT D, CARLINI N, CUBUK E D, et al. Remixmatch: semi-supervised learning with distribution matching and augmentation anchoring [C]// International Conference on Learning Representations. Addis Ababa: [s. n.], 2020: 1-13.
23 BERTHELOT D, CARLINI N, GOODFELLOW I, et al Mixmatch: a holistic approach to semi-supervised learning[J]. Advances in Neural Information Processing Systems, 2019, 32: 155- 166
24 SOHN K, BERTHELOT D, CARLINI N, et al Fixmatch: simplifying semi-supervised learning with consistency and confidence[J]. Advances in Neural Information Processing Systems, 2020, 33: 596- 608
[1] 曾菊香,王平辉,丁益东,兰林,蔡林熹,管晓宏. 面向节点分类的图神经网络节点嵌入增强模型[J]. 浙江大学学报(工学版), 2023, 57(2): 219-225.
[2] 周天琪,杨艳,张继杰,殷少伟,郭增强. 基于无负样本损失和自适应增强的图对比学习[J]. 浙江大学学报(工学版), 2023, 57(2): 259-266.
[3] 王俊陆,李素,纪婉婷,姜天,宋宝燕. 基于Gram矩阵的T-CNN时间序列分类方法[J]. 浙江大学学报(工学版), 2023, 57(2): 267-276.
[4] 张师林,郭红南,刘轩. 基于能量模型的行人与车辆再识别方法[J]. 浙江大学学报(工学版), 2022, 56(7): 1416-1424.
[5] 周国华,卢剑伟,倪彤光,胡学龙. 层次型非线性子空间字典学习[J]. 浙江大学学报(工学版), 2022, 56(6): 1159-1167.
[6] 张雪芹,李天任. 基于Cycle-GAN和改进DPN网络的乳腺癌病理图像分类[J]. 浙江大学学报(工学版), 2022, 56(4): 727-735.
[7] 王婷,朱小飞,唐顾. 基于知识增强的图卷积神经网络的文本分类[J]. 浙江大学学报(工学版), 2022, 56(2): 322-328.
[8] 黄新宇,游帆,张沛,张昭,张柏礼,吕建华,徐立臻. 基于多分类及特征融合的静默活体检测算法[J]. 浙江大学学报(工学版), 2022, 56(2): 263-270.
[9] 李明,段立娟,王文健,恩擎. 基于显著稀疏强关联的脑功能连接分类方法[J]. 浙江大学学报(工学版), 2022, 56(11): 2232-2240.
[10] 彭向东,潘从成,柯泽浚,朱华强,周肖. 基于并行架构和时空注意力机制的心电分类方法[J]. 浙江大学学报(工学版), 2022, 56(10): 1912-1923.
[11] 陈雪云,黄小巧,谢丽. 基于多尺度条件生成对抗网络血细胞图像分类检测方法[J]. 浙江大学学报(工学版), 2021, 55(9): 1772-1781.
[12] 陈智超,焦海宁,杨杰,曾华福. 基于改进MobileNet v2的垃圾图像分类算法[J]. 浙江大学学报(工学版), 2021, 55(8): 1490-1499.
[13] 刘嘉诚,冀俊忠. 基于宽度学习系统的fMRI数据分类方法[J]. 浙江大学学报(工学版), 2021, 55(7): 1270-1278.
[14] 张师林,马思明,顾子谦. 基于大边距度量学习的车辆再识别方法[J]. 浙江大学学报(工学版), 2021, 55(5): 948-956.
[15] 刘芳,汪震,刘睿迪,王锴. 基于组合损失函数的BP神经网络风力发电短期预测方法[J]. 浙江大学学报(工学版), 2021, 55(3): 594-600.