|
|
|
| Video anomaly detection based on multi-scale appearance and motion fusion |
Minghua ZHAO1,2( ),Yuxuan LYU1,Jiahao LYU1(),Yifei CHEN1,Cheng SHI1,Jing HU1 |
1. School of Computer Science and Engineering, Xi’an University of Technology, Xi’an 710048, China
2. Shaanxi Key Laboratory of Network Computing and Security Technology, Xi’an 710048, China |
|
|
|
Abstract A new video anomaly detection method was proposed to address the issue that existing frame prediction-based video anomaly detection methods focus only on appearance features while ignoring motion information and are difficult to simultaneously capture local changes and global anomalies. A dual-stream encoder based on the Inception module was designed to extract multi-scale appearance and optical flow features. The fused features of appearance and motion features at different scales were input into a dynamic prototype unit module to learn the representations of normal features. A hybrid attention mechanism based on CNN-Transformer was added to the skip connection process between the appearance encoder and decoder to realize the integration of global contextual information and local detailed information, therefore enhancing the detection performance. Extensive experiments and ablation studies were conducted on three benchmark datasets: UCSD Ped2, CUHK Avenue, and ShanghaiTech. The AUC values of 99.1%, 88.5%, and 74.5% were achieved, respectively, demonstrating the effectiveness and superiority of the proposed method.
|
|
Received: 11 July 2025
Published: 16 July 2026
|
|
|
| Fund: 陕西省自然科学基金资助项目（2024JC-ZDXM-35, 2024JC-YBMS-573, 2024JC-YBMS-458）；陕西省科协青年人才托举计划资助项目（20240146）；西安理工大学博士创新基金资助项目（BC202621）. |
基于多尺度外观运动融合的视频异常检测
现有的基于帧预测的视频异常检测方法仅关注外观特征而忽略运动信息，难以同时捕捉局部变化与全局异常，为此，提出基于多尺度外观运动融合的视频异常检测方法. 设计基于Inception模块的双流编码器，用于提取多尺度的外观特征与光流特征；将不同尺度外观和运动特征的融合特征输入动态原型单元模块，进行正常特征表征学习；在外观编码器和解码器之间的跳跃连接过程中添加基于CNN-Transformer的混合注意力机制，实现全局上下文信息与局部细节信息的融合，以提升检测性能. 在3个基准数据集UCSD Ped2、CUHK Avenue和ShanghaiTech上进行广泛实验和消融研究，AUC值分别达到了99.1%、88.5%和74.5%，充分验证了所提方法的有效性和优越性.
关键词：
视频异常检测,
卷积神经网络,
时序运动特征,
混合注意力机制,
无监督学习
|
|
| [1] |
GONG D, LIU L, LE V, et al. Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 1705–1714.
|
|
|
| [2] |
RAMACHANDRA B, JONES M J, VATSAVAI R R A survey of single-scene video anomaly detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (5): 2293- 2312
|
|
|
| [3] |
YE M, SHEN J, LIN G, et al Deep learning for person re-identification: a survey and outlook[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (6): 2872- 2893
|
|
|
| [4] |
SABOKROU M, FATHY M, HOSEINI M, et al. Real-time anomaly detection and localization in crowded scenes [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Boston: IEEE, 2015: 56–62.
|
|
|
| [5] |
HASAN M, CHOI J, NEUMANN J, et al. Learning temporal regularity in video sequences [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 733–742.
|
|
|
| [6] |
FENG J, LIANG Y, LI L. Anomaly detection in videos using two-stream autoencoder with post hoc interpretability [J]. Computational Intelligence and Neuroscience, 2021: 7367870.
|
|
|
| [7] |
CHEN D, YUE L, CHANG X, et al NM-GAN: noise-modulated generative adversarial network for video anomaly detection[J]. Pattern Recognition, 2021, 116: 107969
doi: 10.1016/j.patcog.2021.107969
|
|
|
| [8] |
LIU W, LUO W, LIAN D, et al. Future frame prediction for anomaly detection: a new baseline [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6536–6545.
|
|
|
| [9] |
PARK H, NOH J, HAM B. Learning memory-guided normality for anomaly detection [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 14360–14369.
|
|
|
| [10] |
梁家菲, 李婷, 杨佳琪, 等 融合自注意力和自编码器的视频异常检测[J]. 中国图象图形学报, 2023, 28 (4): 1029- 1040
LIANG Jiafei, LI Ting, YANG Jiaqi, et al Video anomaly detection by fusing self-attention and autoencoder[J]. Journal of Image and Graphics, 2023, 28 (4): 1029- 1040
doi: 10.11834/jig.211147
|
|
|
| [11] |
CAI R, ZHANG H, LIU W, et al Appearance-motion memory consistency network for video anomaly detection[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35 (2): 938- 946
doi: 10.1609/aaai.v35i2.16177
|
|
|
| [12] |
LU Y, KUMAR K M, NABAVI S S, et al. Future frame prediction using convolutional VRNN for anomaly detection [C]// Proceedings of the 16th IEEE International Conference on Advanced Video and Signal Based Surveillance. Taipei: IEEE, 2019: 1–8.
|
|
|
| [13] |
WU P, LIU J, LI M, et al Fast sparse coding networks for anomaly detection in videos[J]. Pattern Recognition, 2020, 107: 107515
doi: 10.1016/j.patcog.2020.107515
|
|
|
| [14] |
LU Y, YU F, REDDY M K K, et al. Few-shot scene-adaptive anomaly detection [C]// European Conference on Computer Vision. [S.l.]: Springer, 2020: 125–141.
|
|
|
| [15] |
LYU J, ZHAO M, HU J, et al Bidirectional skip-frame prediction for video anomaly detection with intra-domain disparity-driven attention[J]. Pattern Recognition, 2026, 170: 112010
doi: 10.1016/j.patcog.2025.112010
|
|
|
| [16] |
LI D, NIE X, GONG R, et al Multi-branch GAN-based abnormal events detection via context learning in surveillance videos[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 34 (5): 3439- 3450
|
|
|
| [17] |
LUO W, LIU W, GAO S. A revisit of sparse coding based anomaly detection in stacked RNN framework [C]// Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 341–349.
|
|
|
| [18] |
SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-v4, inception-ResNet and the impact of residual connections on learning [C]// Proceedings of the 31st AAAI Conference on Artificial Intelligence. San Francisco: AAAI Press, 2017: 4278–4284.
|
|
|
| [19] |
LV H, CHEN C, CUI Z, et al. Learning normal dynamics in videos with meta prototype network [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 15420–15429.
|
|
|
| [20] |
LI Z, LI N, JIANG K, et al. Superpixel masking and inpainting for self-supervised anomaly detection [C]// Proceedings of the British Machine Vision Conference. [S.l.]: BMVA Press, 2020.
|
|
|
| [21] |
GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al Generative adversarial networks[J]. Communications of the ACM, 2020, 63 (11): 139- 144
doi: 10.1145/3422622
|
|
|
| [22] |
ZAHEER M Z, LEE J H, MAHMOOD A, et al Stabilizing adversarially learned one-class novelty detection using pseudo anomalies[J]. IEEE Transactions on Image Processing, 2022, 31: 5963- 5975
doi: 10.1109/TIP.2022.3204217
|
|
|
| [23] |
CHEN Y, LIU J, PENG L, et al Auto-encoding variational Bayes[J]. Cambridge Explorations in Arts and Sciences, 2024, 2 (1): 1
|
|
|
| [24] |
GRAVES A, GRAVES A. Long short-term memory [M]// Supervised sequence labelling with recurrent neural networks. Berlin, Heidelberg: Springer, 2012: 37–45.
|
|
|
| [25] |
CHEN D, WANG P, YUE L, et al Anomaly detection in surveillance video based on bidirectional prediction[J]. Image and Vision Computing, 2020, 98: 103915
doi: 10.1016/j.imavis.2020.103915
|
|
|
| [26] |
SUN S, GONG X. Hierarchical semantic contrast for scene-aware video anomaly detection [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 22846–22856.
|
|
|
| [27] |
LIU Z, NIE Y, LONG C, et al. A hybrid video anomaly detection framework via memory-augmented flow reconstruction and flow-guided frame prediction [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 13568–13577.
|
|
|
| [28] |
DOSOVITSKIY A, FISCHER P, ILG E, et al. FlowNet: learning optical flow with convolutional networks [C]// Proceedings of the IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 2758–2766.
|
|
|
| [29] |
ZHONG Y, CHEN X, HU Y, et al Bidirectional spatio-temporal feature learning with multiscale evaluation for video anomaly detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32 (12): 8285- 8296
doi: 10.1109/TCSVT.2022.3190539
|
|
|
| [30] |
MAHADEVAN V, LI W, BHALODIA V, et al. Anomaly detection in crowded scenes [C]// Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco: IEEE, 2010: 1975–1981.
|
|
|
| [31] |
LU C, SHI J, JIA J. Abnormal event detection at 150 FPS in MATLAB [C]// Proceedings of the IEEE International Conference on Computer Vision. Sydney: IEEE, 2013: 2720–2727.
|
|
|
| [32] |
LOSHCHILOV I, HUTTER F. SGDR: stochastic gradient descent with warm restarts [EB/OL]. (2017–05–03) [2025–04–06]. https://arxiv.org/abs/1608.03983.
|
|
|
| [33] |
PASZKE A, GROSS S, CHINTALA S, et al. Automatic differentiation in PyTorch [EB/OL]. (2017–10–29) [2025–04–06]. https://openreview.net/forum?id=BJJsrmfCZ.
|
|
|
| [34] |
ZHANG X, FANG J, YANG B, et al Hybrid attention and motion constraint for anomaly detection in crowded scenes[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33 (5): 2259- 2274
doi: 10.1109/TCSVT.2022.3221622
|
|
|
| [35] |
FANG Z, ZHOU J T, XIAO Y, et al Multi-encoder towards effective anomaly detection in videos[J]. IEEE Transactions on Multimedia, 2020, 23: 4106- 4116
|
|
|
| [36] |
ASTRID M, ZAHEER M Z, LEE J Y, et al. Learning not to reconstruct anomalies [EB/OL]. (2021–10–24) [2025–04–06]. https://arxiv.org/abs/2110.09742.
|
|
|
| [37] |
FAN Y, WEN G, LI D, et al Video anomaly detection and localization via Gaussian mixture fully convolutional variational autoencoder[J]. Computer Vision and Image Understanding, 2020, 195: 102920
doi: 10.1016/j.cviu.2020.102920
|
|
|
| [38] |
WANG B, YANG C Video anomaly detection based on convolutional recurrent AutoEncoder[J]. Sensors, 2022, 22 (12): 4647
doi: 10.3390/s22124647
|
|
|
| [39] |
HUANG C, WEN J, LIU C, et al. Long short-term dynamic prototype alignment learning for video anomaly detection [C]// Proceedings of the 33rd International Joint Conference on Artificial Intelligence. Jeju: Curran Associates Inc, 2024: 866–874.
|
|
|
| [40] |
SUN Z, WANG P, ZHENG W, et al Dual GroupGAN: an unsupervised four-competitor (2V2) approach for video anomaly detection[J]. Pattern Recognition, 2024, 153: 110500
doi: 10.1016/j.patcog.2024.110500
|
|
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
