基于光流重投影的高性能轻量级帧外插技术

doi:10.3785/j.issn.1008-973X.2025.05.003

浙江大学学报(工学版)

2025, Vol. 59

Issue (5): 902-911 DOI: 10.3785/j.issn.1008-973X.2025.05.003

计算机技术、信息工程

基于光流重投影的高性能轻量级帧外插技术

覃浩宇(

),过洁*(

),张浩南,冯泽森,浦亮,张嘉伟,郭延文

南京大学计算机科学与技术系，江苏南京 210033

High-performance lightweight frame extrapolation technique based on optical flow reprojection

Haoyu QIN(

),Jie GUO*(

),Haonan ZHANG,Zesen FENG,Liang PU,Jiawei ZHANG,Yanwen GUO

Department of Computer Science and Technology, Nanjing University, Nanjing 210033, China

全文: PDF(1410 KB) HTML

摘要：

为了解决实时渲染过程时间开销较大的问题，提出基于光流重投影的高性能轻量级帧外插技术.提出光流重投影模块，基于深度学习预测相邻帧之间的光流信息，对历史帧执行光流重投影，解决了阴影、高光区域没有运动信息的问题. 该方法引入多种轻量化设计，包括低分辨率网络推理、基于SE模块的图像填补，大幅减少了方法的时间开销. 经过实验验证可知，相对于最前沿的帧外插技术，该方法能够达到3倍的时间性能提升.

关键词： 实时渲染; 帧外插; 光流重投影; 图像填补

Abstract:

A high-performance lightweight frame extrapolation technique based on optical flow reprojection was proposed in order to solve the problem of high time overhead in real-time rendering process. An optical flow reprojection module was proposed, which predicted the optical flow information between adjacent frames based on deep learning and applied optical flow reprojection to historical frames. Then the issue of areas such as shadows and highlights lacking motion information was addressed. The proposed method incorporated various lightweight designs, including low-resolution network inference and image inpainting based on SE (squeeze-and-excitation) modules, significantly reducing the computational overhead. The experimental results demonstrate that the proposed method achieves a threefold improvement in time performance compared to state-of-the-art frame extrapolation techniques.

Key words: real-time rendering frame extrapolation optical flow reprojection image inpainting

收稿日期: 2024-07-01 出版日期: 2025-04-25

CLC:

TP 391

基金资助: 中央高校基本科研业务费专项资金资助项目(2024300326).

通讯作者: 过洁 E-mail: 449139777@qq.com;guojie@nju.edu.cn

作者简介: 覃浩宇（1998—），男，硕士生，从事计算机图形学的研究. orcid.org/0009-0006-0970-9659. E-mail：449139777@qq.com

	服务
	把本文推荐给朋友
	加入引用管理器
	E-mail Alert
	作者相关文章
	覃浩宇
	过洁
	张浩南
	冯泽森
	浦亮
	张嘉伟
	郭延文

引用本文:

覃浩宇,过洁,张浩南,冯泽森,浦亮,张嘉伟,郭延文. 基于光流重投影的高性能轻量级帧外插技术[J]. 浙江大学学报(工学版), 2025, 59(5): 902-911.

Haoyu QIN,Jie GUO,Haonan ZHANG,Zesen FENG,Liang PU,Jiawei ZHANG,Yanwen GUO. High-performance lightweight frame extrapolation technique based on optical flow reprojection. Journal of ZheJiang University (Engineering Science), 2025, 59(5): 902-911.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2025.05.003 或 https://www.zjujournals.com/eng/CN/Y2025/V59/I5/902

图 1 帧外插网络的总框架

图 2 图像修补模块

图 3 门卷积模块

图 4 SE模块

图 5 光流重投影模块

表 1 提出方法与ExtraNet的时间开销/模型规模对比

图 6 Medieval Docks( 1、2 排) 和Redwood Forest( 3、4 排) 场景的帧外插结果对比

表 2 不同场景下的定量指标结果对比

表 3 模块替换前、后的实验指标结果

表 4 不同G-buffers选取策略的实验指标结果

图 7 G-buffers重要性实验结果

图 8 光流重投影模块消融实验的结果

图 9 SE模块消融实验的结果

1	SANDY M, ANDERSSON J, BARRÉ-BRISEBOIS C. Directx: evolving microsoft’s graphics platform [C]// Game Developers Conference . San Francisco: IEEE, 2018.
2	BURGESS J RTX on the NVIDIA Turing GPU[J]. IEEE Micro, 2020, 40 (2): 36- 44 doi: 10.1109/MM.2020.2971677
3	HARADA T. Hardware-accelerated ray tracing in AMD Radeon ProRender 2.0 [EB/OL]. [2024-06-20]. https://gpuopen.com/learn/radeon-prorender-2-0/.
4	GUO J, FU X, LIN L, et al. ExtraNet: real-time extrapolated rendering for low-latency temporal supersampling [J]. ACM Transactions on Graphics , 2021, 40(6): 1-16.
5	BAO W, LAI W S, MA C, et al. Depth-aware video frame interpolation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Los Angeles: IEEE, 2019.
6	LEE H, KIM T, CHUNG T Y, et al. AdaCoF: adaptive collaboration of flows for video frame interpolation [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle: IEEE, 2020.
7	NIKLAUS S, MAI L, LIU F. Video frame interpolation via adaptive convolution [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Hawaii: IEEE, 2017.
8	OCULUS V. Asynchronous spacewarp [EB/OL]. [2024-06-20]. https://developers.meta.com/horizon/blog/asynchronous-spacewarp/.
9	MEYER S, WANG O, ZIMMER H, et al. Phase-based frame interpolation for video [C]// IEEE Conference on Computer Vision and Pattern Recognition .Washington DC: IEEE, 2015: 1410-1418.
10	MEYER S, DJELOUAH A, MCWILLIAMS B, et al. PhaseNet for video frame interpolation [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition . Los Alamitos: IEEE, 2018: 498-507.
11	JIANG H, SUN D, JAMPANI V, et al. Super SloMo: high quality estimation of multiple intermediate frames for video interpolation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Los Alamitos: IEEE, 2018.
12	LONG G, KNEIP L, ALVAREZ J M, et al. Learning image matching by simply watching video [C]//LEIBE B, MATAS J, SEBE N, et al. European Conference on Computer Vision . Cham: Springer, 2016: 434-450.
13	CHOI M, KIM H, HAN B, et al. Channel attention is all you need for video frame interpolation [C]// AAAI Conference on Artificial Intelligence . New York: AAAI, 2020.
14	KALLURI T, PATHAK D, CHANDRAKER M, et al. FLAVR: flow-agnostic video representations for fast frame interpolation [C]// IEEE Workshop/Winter Conference on Applications of Computer Vision . [S. l.]: IEEE, 2020.
15	LU L, WU R, LIN H, et al. Video frame interpolation with Transformer [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . New Orleans: IEEE, 2022: 3532-3542.
16	REDA F, KONTKANEN J, TABELLION E, et al. FILM: frame interpolation for large motion [C]// European Conference on Computer Vision . Tel Aviv: Springer, 2022.
17	DIDYK P, EISEMANN E, RITSCHEL T, et al Perceptually-motivated real-time temporal upsampling of 3D content for high-refresh-rate displays[J]. Computer Graphics Forum, 2010, 29 (2): 713- 722 doi: 10.1111/j.1467-8659.2009.01641.x
18	DIDYK P, RITSCHEL T, EISEMANN E, et al. Adaptive image-space stereo view synthesis [C]// Proceedings of Vision, Modeling, and Visualization Workshop 2010 . Siegen: [s. n.], 2010.
19	YANG L, TSE Y C, SANDER P V, et al. Image-based bidirectional scene reprojection [C]// Proceedings of the 2011 SIGGRAPH Asia Conference . Hongkong: ACM, 2011: 1-10.
20	BOWLES H, MITCHELL K, SUMNER R, et al. Iterative image warping [C]// Computer Graphics Forum. Oxford: Blackwell Publishing Ltd, 2012: 237-246.
21	NTAVELIS E, ROMERO A, BIGDELI S, et al. AIM 2020 challenge on image extreme inpainting [EB/OL]. [2024-06-20]. http://arxiv.org/abs/2010.01110.
22	PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting [EB/OL]. [2024-06-20]. https://arxiv.org/abs/1604.07379.
23	ZENG Y, FU J, CHAO H, et al. Learning pyramid-context encoder network for high-quality image inpainting [EB/OL]. [2024-06-20]. https://arxiv.org/abs/1904.07475.
24	LIU G, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions [EB/OL]. [2024-06-20]. https://arxiv.org/abs/1804.07723.
25	YU J, LIN Z, YANG J, et al. Free-form image inpainting with gated convolution [EB/OL]. [2024-06-20]. https://arxiv.org/abs/1806.03589.
26	YU F, KOLTUN V. Multi-scale context aggregation by dilated convolutions [EB/OL]. [2024-06-20]. https://arxiv.org/abs/1511.07122.
27	REN Y, YU X, ZHANG R, et al. StructureFlow: image inpainting via structure-aware appearance flow [EB/OL]. [2024-06-20]. https://arxiv.org/abs/1908.03852.
28	LIU H, WANG Y, WANG M, et al. Delving globally into texture and structure for image inpainting [C]// Proceedings of the 30th ACM International Conference on Multimedia . [S. l. ]: ACM, 2022.
29	PIRNAY J, CHAI K. Inpainting Transformer for anomaly detection [EB/OL]. [2024-06-20]. https://arxiv.org/abs/2104.13897.
30	LUGMAYR A, DANELLJAN M, ROMERO A, et al. RePaint: inpainting using denoising diffusion probabilistic models [EB/OL]. [2024-06-20]. https://arxiv.org/abs/2201.09865.
31	HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models [[EB/OL]. [2024-06-20]. https://arxiv.org/abs/2006.11239.
32	GUO J, LAI S, TAO C, et al. Highlight-aware two-stream network for single image SVBRDF acquisition [J]. ACM Transactions on Graphics , 2021, 40(4): 1-14.
33	ZENG Z, LIU S, YANG J, et al Temporally reliable motion vectors for real-time ray tracing[J]. Computer Graphics Forum, 2021, 40 (2): 79- 90 doi: 10.1111/cgf.142616
34	HU J, SHEN L, SUN G. Squeeze-and-excitation networks [C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition . Los Alamitos: IEEE, 2018: 7132-7141.
35	PASZKE A, GROSS S, MASSA F, et al. PyTorch: an imperative style, high-performance deep learning library [C]//WALLACH H M, LAROCHELLE H, BEYGELZIMER A, et al. Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver: MIT Press, 2019: 8024-8035.
36	KINGMA D P, BA J. Adam: a method for stochastic optimization [C]// BENGIO Y, LECUN Y. 3rd International Conference on Learning Representations. San Diego: MIT Press, 2015.
37	SCHIED C, KAPLANYAN A, WYMAN C, et al. Spatiotemporal variance-guided filtering: real-time reconstruction for path-traced global illumination [C]// Proceedings of High Performance Graphics .[S. l.]: ACM, 2017 .

[1]	张自然,李锵,关欣. 基于卷积辅助自注意力的胸部疾病分类网络[J]. 浙江大学学报(工学版), 2025, 59(5): 890-901.
[2]	李宗民,徐畅,白云,鲜世洋,戎光彩. 面向点云理解的双邻域图卷积方法[J]. 浙江大学学报(工学版), 2025, 59(5): 879-889.
[3]	董镇滔,徐暟敏,万清颖,刘晓菲,申昊,李书涵,奇格奇. 基于交通事件短视频资源的多模态情绪特征分析[J]. 浙江大学学报(工学版), 2025, 59(4): 661-668.
[4]	李沈崇,曾新华,林传渠. 基于轴向注意力的多任务自动驾驶环境感知算法[J]. 浙江大学学报(工学版), 2025, 59(4): 769-777.
[5]	顾正宇,赖菲菲,耿辰,王希明,戴亚康. 基于知识引导的缺血性脑卒中梗死区分割方法[J]. 浙江大学学报(工学版), 2025, 59(4): 814-820.
[6]	刘登峰,郭文静,陈世海. 基于内容引导注意力的车道线检测网络[J]. 浙江大学学报(工学版), 2025, 59(3): 451-459.
[7]	姚明辉,王悦燕,吴启亮,牛燕,王聪. 基于小样本人体运动行为识别的孪生网络算法[J]. 浙江大学学报(工学版), 2025, 59(3): 504-511.
[8]	梁礼明,龙鹏威,金家新,李仁杰,曾璐. 基于改进YOLOv8s的钢材表面缺陷检测算法[J]. 浙江大学学报(工学版), 2025, 59(3): 512-522.
[9]	王浚银,文斌,沈艳军,张俊,王子豪. 基于改进YOLOv7-tiny的铝型材表面缺陷检测方法[J]. 浙江大学学报(工学版), 2025, 59(3): 523-534.
[10]	尹向雷,屈少鹏,解永芳,苏妮. 基于渐进特征融合及多尺度空洞注意力的遮挡鸟巢检测[J]. 浙江大学学报(工学版), 2025, 59(3): 535-545.
[11]	杨凯博,钟铭恩,谭佳威,邓智颖,周梦丽,肖子佶. 基于半监督学习的多场景火灾小规模稀薄烟雾检测[J]. 浙江大学学报(工学版), 2025, 59(3): 546-556.
[12]	高铭宇,徐敬华,张树有,王康,谭建荣. 基于邻域拓扑重建的人体工学产品定制设计方法[J]. 浙江大学学报(工学版), 2025, 59(3): 597-605.
[13]	李颂元,朱祥维,李玺. 基座模型技术背景下的具身智能体综述[J]. 浙江大学学报(工学版), 2025, 59(2): 213-226.
[14]	董红召,林少轩,佘翊妮. 交通目标YOLO检测技术的研究进展[J]. 浙江大学学报(工学版), 2025, 59(2): 249-260.
[15]	薛雅丽,贺怡铭,崔闪,欧阳权. 基于改进YOLOv5的SAR图像有向舰船目标检测算法[J]. 浙江大学学报(工学版), 2025, 59(2): 261-268.

Viewed

Full text

Abstract

Cited

Shared

Discussed