Please wait a minute...
Journal of ZheJiang University (Engineering Science)  2023, Vol. 57 Issue (4): 744-752    DOI: 10.3785/j.issn.1008-973X.2023.04.012
    
Semi-supervised learning method based on distance metric loss framework
Ban-teng LIU1,2(),Zan-ting YE2,Hai-long QIN3,Ke WANG1,4,*(),Qi-hang ZHENG1,Zhang-quan WANG1,2
1. College of Information Science and Technology, Zhejiang Shuren University, Hangzhou 310015, China
2. College of Computer Science and Artificial Intelligence, Changzhou University, Changzhou 213164, China
3. Zhejiang Lvcheng Future Digital Intelligence Technology Limited Company, Hangzhou 311121, China
4. State Key Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou 310027, China
Download: HTML     PDF(1146KB) HTML
Export: BibTeX | EndNote (RIS)      

Abstract  

A semi-supervised learning method based on the distance metric loss framework was proposed in order to solve the problems of different types of loss functions and inconsistent loss scales in the training process of semi-supervised learning methods, which make it difficult to adjust the loss weights, inconsistent optimization directions and insufficient generalization ability. A unify loss framework function was proposed from the perspective of distance metric loss, and the adjustment of loss weights between different loss functions in semi-supervised tasks was achieved. Adaptive similarity weights were introduced for the target region problem of embedding vectors in the loss framework in order to avoid the conflict of optimization directions of traditional metric learning loss functions and improve the generalization performance of the model. CNN13 and ResNet18 networks were used to construct semi-supervised learning models on CIFAR-10, CIFAR-100, SVHN, STL-10 standard image dataset and medical pneumonia dataset Pneumonia Chest X-ray, respectively, for comparison with commonly used semi-supervised methods in order to validate the effectiveness of the method. Results show that the method has the optimal classification accuracy under the condition of the same number of labels.



Key wordssemi-supervised learning      metric learning      loss function      loss framework      classification     
Received: 07 April 2022      Published: 21 April 2023
CLC:  TP 391  
Fund:  浙江省“领雁”研发攻关计划资助项目(2022C03122);浙江省公益技术应用研究资助项目(LGF22F020006,LGF21F010004);浙江大学工业控制技术国家重点实验室开放课题资助项目(ICT2022B34)
Corresponding Authors: Ke WANG     E-mail: hupo3@sina.com;wangke1992@zju.edu.cn
Cite this article:

Ban-teng LIU,Zan-ting YE,Hai-long QIN,Ke WANG,Qi-hang ZHENG,Zhang-quan WANG. Semi-supervised learning method based on distance metric loss framework. Journal of ZheJiang University (Engineering Science), 2023, 57(4): 744-752.

URL:

https://www.zjujournals.com/eng/10.3785/j.issn.1008-973X.2023.04.012     OR     https://www.zjujournals.com/eng/Y2023/V57/I4/744


基于距离度量损失框架的半监督学习方法

为了解决半监督学习方法训练过程中因损失函数类型不同、损失尺度不统一而导致的损失权重难以调节, 模型优化方向不统一与泛化能力不足的问题, 提出基于距离度量损失框架的半监督学习方法. 该方法从距离度量损失的角度出发, 提出统一损失框架函数, 实现了半监督任务中不同损失函数之间的损失权重调节. 针对损失框架中嵌入向量的目标区域问题, 引入自适应相似度权重,以避免传统度量学习损失函数优化方向的冲突, 提高模型的泛化性能. 为了验证方法的有效性, 分别采用CNN13网络和ResNet18网络,在CIFAR-10、CIFAR-100、SVHN、STL-10标准图像数据集和医疗肺炎数据集Pneumonia Chest X-ray上,构建半监督学习模型与常用半监督方法进行比较. 实验结果表明, 在同等标签数目的条件下, 提出方法具有最优的分类准确度.


关键词: 半监督学习,  度量学习,  损失函数,  损失框架,  分类 
Fig.1 Schematic diagram of entropy minimization and consistency loss
Fig.2 Schematic diagram of optimization method of embedd-ing vector and feature vector
Fig.3 Diagram of Unify Loss function
Fig.4 Contrast diagram of loss gradient
数据集 N Nt
Nl Nu
SVHN 40 70 000 26 000
250 70 000
4 000 70 000
CIFAR-10 40 50 000 5 000
250 50 000
4 000 50 000
CIFAR-100 400 40 000 10 000
1 000 40 000
10 000 40 000
STL-10 1 000 10 000 8 000
5 000 10 000
Pneumonia Chest X-ray 250 4 750 600
500 4 500
1 000 4 000
Tab.1 Training and test sample distribution of each dataset
Fig.5 Comparison diagram of each parameter value
方法 CIFAR-10 CIFAR-100 SVHN STL-10
Nl = 40 Nl = 250 Nl = 4000 Nl = 400 Nl = 1000 Nl = 10000 Nl = 40 Nl = 250 Nl = 4000 Nl = 1000 Nl = 5000
VAT 27.41 53.97 82.59 5.12 12.89 53.20 35.64 79.78 90.06 68.77 81.26
MeanTeacher 26.54 55.52 83.79 4.94 12.13 55.13 36.45 78.43 93.58 69.01 82.80
MixMatch 41.47 74.08 90.74 9.15 32.39 65.87 47.45 76.02 93.50 79.59 88.41
Remixmatch 42.32 78.71 92.20 12.98 44.56 72.33 52.55 81.92 94.67 84.33 92.89
UDA 44.55 81.66 92.45 10.66 40.72 71.18 47.37 84.31 95.54 82.34 92.74
UDA_unify 49.69 84.22 93.87 18.06 46.12 72.67 53.14 87.32 95.77 87.21 94.34
Fixmatch 50.17 86.28 93.11 12.65 42.95 72.08 56.41 88.75 95.94 89.68 94.58
Fixmatch_unify 55.24 89.94 94.74 18.51 47.79 73.38 65.86 90.09 96.02 93.44 95.25
Flexmatch 52.70 85.08 93.97 27.22 55.12 77.84 62.03 90.42 95.83 92.36 96.06
Unify Loss 61.19 90.33 94.25 35.35 61.75 78.11 67.35 92.47 96.17 93.42 96.51
Tab.2 Accuracy of each method on CIFAR-10, CIFAR-100, SVHN, STL-10 datasets
方法 A/%
Nl/N = 5% Nl/N = 10% Nl/N = 20%
VAT 8.62 25.71 67.12
MeanTeacher 10.72 27.92 69.46
MixMatch 12.24 31.62 73.86
Remixmatch 15.07 33.78 74.02
UDA 11.87 31.89 75.31
UDA_unify 17.66 36.46 76.55
Fixmatch 19.12 37.22 77.54
Fixmatch_unify 23.55 40.91 77.67
Flexmatch 23.38 40.67 78.76
Unify Loss 26.57 42.26 79.24
Tab.3 Validation results of each method on Pneumonia Chest X-ray dataset
[1]   KORNBLITH S, SHLENS J, LE Q V. Do better imagenet models transfer better? [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 2661-2671.
[2]   YANG S, LUO P, LOY C C, et al. WIDER FACE: a face detection benchmark [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 5525-5533.
[3]   许佳辉, 王敬昌, 陈岭, 等 基于图神经网络的地表水水质预测模型[J]. 浙江大学学报:工学版, 2021, 55 (4): 601- 607
XU Jia-hui, WANG Jing-chang, CHEN Ling, et al Surface water quality prediction model based on graph neural network[J]. Journal of Zhejiang University: Engineering Science, 2021, 55 (4): 601- 607
[4]   LEE D H. Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks [C]// ICML 2013 Workshop on Challenges in Representation Learning. Atlanta: PMLR, 2013: 896.
[5]   TARVAINEN A, VALPOLA H Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results[J]. Advances in Neural Information Processing Systems, 2017, 30: 1195- 1204
[6]   XIE Q, DAI Z, HOVV E, et al Unsupervised data augmentation for consistency training[J]. Advances in Neural Information Processing Systems, 2020, 33 (2): 6256- 6268
[7]   MIYATO T, MAEDA S, KOYAMA M, et al Virtual adversarial training: a regularization method for supervised and semi-supervised learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41 (8): 1979- 1993
[8]   WANG F, CHENG J, LIU W, et al Additive margin softmax for face verification[J]. IEEE Signal Processing Letters, 2018, 25 (7): 926- 930
doi: 10.1109/LSP.2018.2822810
[9]   LAINE S, AILA T. Temporal ensembling for semi-supervised learning [C]// International Conference on Learning Representations. Toulon: [s. n.], 2017: 1-13.
[10]   SAJJADI M, JAVANMARDI M, TASDIZEN T Regularization with stochastic transformations and perturbations for deep semi-supervised learning[J]. Advances in Neural Information Processing Systems, 2016, 29 (7): 1163- 1171
[11]   LIU W, WEN Y, YU Z, et al. Large-margin softmax loss for convolutional neural networks [C]// Proceedings of the 33rd International Conference on Machine Learning. New York: PMLR, 2016: 507-516.
[12]   LI Y, GAO F, OU Z, et al. Angular softmax loss for end-to-end speaker verification [C]// 2018 11th International Symposium on Chinese Spoken Language Processing. Taipei: IEEE, 2018: 190-194.
[13]   GRANDVALET Y, BENGIO Y Semi-supervised learning by entropy minimization[J]. Advances in Neural Information Processing Systems, 2004, 17: 529- 536
[14]   VERMA V, KAWAGUCHI K, LAMB A, et al Interpolation consistency training for semi-supervised learning[J]. Neural Networks, 2022, 145: 90- 106
doi: 10.1016/j.neunet.2021.10.008
[15]   HENDRYCKS D, MU N, CUBUK E D, et al. Augmix: a simple method to improve robustness and uncertainty under data shift [C]// International Conference on Learning Representations. Ethiopia: [s. n.], 2020: 1-15.
[16]   KENDALL A, GAL Y, CIPOLLA R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7482-7491.
[17]   AZIERE N, TODOROVIC S. Ensemble deep manifold similarity learning using hard proxies [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 7299-7307.
[18]   QIAN Q, SHANG L, SUN B, et al. Softtriple loss: deep metric learning without triplet sampling [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 6450-6458.
[19]   KIM S, KIM D, CHO M, et al. Proxy anchor loss for deep metric learning [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 3238-3247.
[20]   SUN Y, CHENG C, ZHANG Y, et al. Circle loss: a unified perspective of pair similarity optimization [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 6398-6407.
[21]   KERMANY D, ZHANG K, GOLDBAUM M Labeled optical coherence tomography (oct) and chest X-ray images for classification[J]. Mendeley Data, 2018, 2 (2): 255- 265
[22]   BERTHELOT D, CARLINI N, CUBUK E D, et al. Remixmatch: semi-supervised learning with distribution matching and augmentation anchoring [C]// International Conference on Learning Representations. Addis Ababa: [s. n.], 2020: 1-13.
[23]   BERTHELOT D, CARLINI N, GOODFELLOW I, et al Mixmatch: a holistic approach to semi-supervised learning[J]. Advances in Neural Information Processing Systems, 2019, 32: 155- 166
[24]   SOHN K, BERTHELOT D, CARLINI N, et al Fixmatch: simplifying semi-supervised learning with consistency and confidence[J]. Advances in Neural Information Processing Systems, 2020, 33: 596- 608
[1] Ju-xiang ZENG,Ping-hui WANG,Yi-dong DING,Lin LAN,Lin-xi CAI,Xiao-hong GUAN. Graph neural network based node embedding enhancement model for node classification[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(2): 219-225.
[2] Tian-qi ZHOU,Yan YANG,Ji-jie ZHANG,Shao-wei YIN,Zeng-qiang GUO. Graph contrastive learning based on negative-sample-free loss and adaptive augmentation[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(2): 259-266.
[3] Jun-lu WANG,Su LI,Wan-ting JI,Tian JIANG,Bao-yan SONG. T-CNN time series classification method based on Gram matrix[J]. Journal of ZheJiang University (Engineering Science), 2023, 57(2): 267-276.
[4] Shi-lin ZHANG,Hong-nan GUO,Xuan LIU. Person and vehicle re-identification based on energy model[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(7): 1416-1424.
[5] Guo-hua ZHOU,Jian-wei LU,Tong-guang NI,Xue-long HU. Hierarchical nonlinear subspace dictionary learning[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(6): 1159-1167.
[6] Xue-qin ZHANG,Tian-ren LI. Breast cancer pathological image classification based on Cycle-GAN and improved DPN network[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(4): 727-735.
[7] Ting WANG,Xiao-fei ZHU,Gu TANG. Knowledge-enhanced graph convolutional neural networks for text classification[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 322-328.
[8] Xin-yu HUANG,Fan YOU,Pei ZHANG,Zhao ZHANG,Bai-li ZHANG,Jian-hua LV,Li-zhen XU. Silent liveness detection algorithm based on multi classification and feature fusion network[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(2): 263-270.
[9] Ming LI,Li-juan DUAN,Wen-jian WANG,Qing EN. Brain functional connections classification method based on significant sparse strong correlation[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(11): 2232-2240.
[10] Xiang-dong PENG,Cong-cheng PAN,Ze-jun KE,Hua-qiang ZHU,Xiao ZHOU. Classification method for electrocardiograph signals based on parallel architecture model and spatial-temporal attention mechanism[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(10): 1912-1923.
[11] Xiao-feng FU,Li NIU. Micro-expression classification based on deep convolution and auto-encoder enhancement[J]. Journal of ZheJiang University (Engineering Science), 2022, 56(10): 1948-1957.
[12] Xue-yun CHEN,Xiao-qiao HUANG,Li XIE. Classification and detection method of blood cells images based on multi-scale conditional generative adversarial network[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(9): 1772-1781.
[13] Zhi-chao CHEN,Hai-ning JIAO,Jie YANG,Hua-fu ZENG. Garbage image classification algorithm based on improved MobileNet v2[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(8): 1490-1499.
[14] Jia-cheng LIU,Jun-zhong JI. Classification method of fMRI data based on broad learning system[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(7): 1270-1278.
[15] Shi-lin ZHANG,Si-ming MA,Zi-qian GU. Large margin metric learning based vehicle re-identification method[J]. Journal of ZheJiang University (Engineering Science), 2021, 55(5): 948-956.