基于知识图谱引导的三维视觉问答

doi:10.3785/j.issn.1008-973X.2026.08.019

浙江大学学报(工学版)

2026, Vol. 60

Issue (8): 1801-1808 DOI: 10.3785/j.issn.1008-973X.2026.08.019

计算机技术

基于知识图谱引导的三维视觉问答

毛爱华(

),陈思宇

华南理工大学计算机科学与工程学院，广东广州 511400

3D visual question answering guided by knowledge graph

Aihua MAO(

),Siyu CHEN

School of Computer Science and Engineering, South China University of Technology, Guangzhou 511400, China

全文: PDF(2447 KB) HTML

摘要：

针对场景中物体之间潜在的常识性语义关系，提出基于知识图谱引导的三维视觉问答方法，通过引入外部结构化知识，有效增强模型的语义理解与推理能力. 从问题文本中提取关键语义实体，设计知识图谱引导的特征增强模块，利用关键语义实体获取知识特征. 将知识特征与由问题表示和三维目标检测网络提取的视觉特征进行融合，用于答案预测. 在ScanQA数据集上的实验证明，所提方法在评估指标EM@1与BLEU-4上均优于现有基线模型.

关键词： 三维视觉问答; 知识图谱; 特征增强; 多模态融合; 语义关系; 先验信息

Abstract:

A knowledge graph-guided 3D visual question answering method was proposed to capture the implicit common-sense semantic relationships between objects in the scene. By introducing external structured knowledge, the model was effectively enhanced in both semantic understanding and reasoning ability. Specifically, key semantic entities were extracted from the question text, and a knowledge graph-guided feature enhancement module was designed to obtain knowledge features using these key semantic entities. The knowledge features were fused with visual features extracted both from the question representation and from the 3D object detection network for answer prediction. Experimental results on the ScanQA dataset showed that the proposed method outperforms existing baseline models on metrics such as EM@1 and BLEU-4.

Key words: 3D visual question answering knowledge graph feature enhancement multimodal fusion semantic relationship prior information

收稿日期: 2025-07-10 出版日期: 2026-07-16

CLC:

TP 242.6

基金资助: 广东省基础与应用基础研究基金资助项目（2024A1515012791）.

作者简介: 毛爱华（1980—），男，教授，博导，博士，从事人工智能、三维视觉研究. orcid.org/0000-0001-6861-9414. E-mail：ahmao@scut.edu.cn

	服务
	把本文推荐给朋友
	加入引用管理器
	E-mail Alert
	作者相关文章
	毛爱华
	陈思宇

引用本文:

毛爱华,陈思宇. 基于知识图谱引导的三维视觉问答[J]. 浙江大学学报(工学版), 2026, 60(8): 1801-1808.

Aihua MAO,Siyu CHEN. 3D visual question answering guided by knowledge graph. Journal of ZheJiang University (Engineering Science), 2026, 60(8): 1801-1808.

链接本文:

https://www.zjujournals.com/eng/CN/10.3785/j.issn.1008-973X.2026.08.019 或 https://www.zjujournals.com/eng/CN/Y2026/V60/I8/1801

图 1 基于知识图谱引导的三维视觉问答示意图

图 2 知识图谱引导的特征增强模块结构

图 3 多模态融合模块结构

表 1 不同方法在ScanQA带有对象的测试集上的评估指标结果对比

表 2 不同方法在ScanQA不带对象的测试集上的评估指标结果对比

图 4 基于知识图谱引导的三维视觉问答方法在ScanQA数据集上的可视化结果

表 3 不同模型变体在ScanQA带对象的测试集上的评估指标结果对比

1	AZUMA D, MIYANISHI T, KURITA S, et al. ScanQA: 3D question answering for spatial scene understanding [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 19107–19117.
2	DELITZAS A, PARELLI M, HARS N, et al. Multi-CLIP: contrastive vision-language pre-training for question answering tasks in 3D scenes [EB/OL]. (2023–06–04)[2025–07–02]. https://arxiv.org/pdf/2306.02329.
3	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision [EB/OL]. (2021–02–26)[2025–07–02]. https://arxiv.org/pdf/2103.00020.
4	JIN Z, HAYAT M, YANG Y, et al. Context-aware alignment and mutual masking for 3D-language pre-training [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 10984–10994.
5	CHEN D Z, CHANG A X, NIEßNER M. ScanRefer: 3D object localization in RGB-D scans using natural language [C]// Computer Vision – ECCV 2020. [S.l.]: Springer International Publishing, 2020: 202–221.
6	HONG Y, ZHEN H, CHEN P, et al. 3D-LLM: injecting the 3D world into large language models [EB/OL]. (2023–07–24)[2025–07–02]. https://arxiv.org/pdf/2307.12981.
7	LI J, LI D, SAVARESE S, et al. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models [C]// Proceedings of the International Conference on Machine Learning. [S.l.]: PMLR, 2023: 19730-19742.
8	ALAYRAC J B, DONAHUE J, LUC P, et al. Flamingo: a visual language model for few-shot learning [EB/OL]. (2022–11–15)[2025–07–02]. https://arxiv.org/pdf/2204.14198.
9	PAPINENI K, ROUKOS S, WARD T, et al. BLEU: a method for automatic evaluation of machine translation [C]// Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Philadelphia: Association for Computational Linguistics, 2002: 311–318.
10	VEDANTAM R, ZITNICK C L, PARIKH D. CIDEr: consensus-based image description evaluation [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 4566–4575.
11	ZHU Z, MA X, CHEN Y, et al. 3D-VisTA: pre-trained transformer for 3D vision and text alignment [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2024: 2899–2909.
12	VO D M, CHEN H, SUGIMOTO A, et al. NOC-REK: novel object captioning with retrieved vocabulary from external knowledge [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 17979–17987.
13	SPEER R, CHIN J, HAVASI C ConceptNet 5.5: an open multilingual graph of general knowledge[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2017, 31 (1): 1 doi: 10.1609/aaai.v31i1.11164
14	DING Z, HAN X, NIETHAMMER M. VoteNet: a deep learning label fusion method for multi-atlas segmentation [C]// Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. [S.l.]: Springer, 2019: 202–210.
15	PENNINGTON J, SOCHER R, MANNING C. GloVe: global vectors for word representation [C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Doha: Association for Computational Linguistics, 2014: 1532–1543.
16	SCHUSTER M, PALIWAL K K Bidirectional recurrent neural networks[J]. IEEE Transactions on Signal Processing, 1997, 45 (11): 2673- 2681 doi: 10.1109/78.650093
17	KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks [EB/OL]. (2017–02–22)[2025–07–02]. https://arxiv.org/pdf/1609.02907.
18	RUMELHART D E, HINTON G E, WILLIAMS R J Learning representations by back-propagating errors[J]. Nature, 1986, 323 (6088): 533- 536 doi: 10.1038/323533a0
19	LIN C Y, HOVY E. Automatic evaluation of summaries using N-gram co-occurrence statistics [C]// Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03. Edmonton: ACL, 2003: 71–78.

[1]	吴佰靖,闫光辉,马龙,程文鑫,黄亚宁. 边缘感知和跨尺度特征增强的小目标水漂垃圾检测[J]. 浙江大学学报(工学版), 2026, 60(7): 1452-1463.
[2]	高一聪,吴栋,密尚华,郑浩,谭建荣. 思维链增强的机电装备运维方案智能生成方法[J]. 浙江大学学报(工学版), 2026, 60(7): 1515-1527.
[3]	宋耀莲,彭驰,唐菁敏,赵宣植,虞贵财. 基于融合注意力机制的光学遥感图像小目标检测算法[J]. 浙江大学学报(工学版), 2026, 60(4): 763-771.
[4]	张雪梅,孙颖,张雪英. 基于无监督图对比学习的语音情感识别[J]. 浙江大学学报(工学版), 2026, 60(4): 782-790.
[5]	冯超文,耿程晨,刘英莉. 基于嵌入特征和稀疏矩阵的实体对齐方法[J]. 浙江大学学报(工学版), 2026, 60(2): 379-387.
[6]	谢涛,葛慧丽,陈宁,汪晓锋,李延松,黄晓峰. 知识嵌入增强的对比推荐模型[J]. 浙江大学学报(工学版), 2026, 60(1): 90-98.
[7]	黄文湖,赵邢,谢亮,梁浩然,梁荣华. 基于对比学习的声源定位引导视听分割模型[J]. 浙江大学学报(工学版), 2025, 59(9): 1803-1813.
[8]	周昶清,侯耀春,武鹏,杨帅,吴大转. 自适应齿轮箱稀疏表示原子构建方法[J]. 浙江大学学报(工学版), 2025, 59(5): 1018-1030.
[9]	杨燕,贾存鹏. 代理注意力下域特征交互的高效图像去雾算法[J]. 浙江大学学报(工学版), 2025, 59(12): 2527-2538.
[10]	陈智超,杨杰,李凡,冯志成. 基于深度学习的列车运行环境感知关键算法研究综述[J]. 浙江大学学报(工学版), 2025, 59(1): 1-17.
[11]	李忠良,陈麒,石琳,杨朝,邹先明. 时间感知组合的动态知识图谱补全[J]. 浙江大学学报(工学版), 2024, 58(8): 1738-1747.
[12]	李劲业,李永强. 融合知识图谱的时空多图卷积交通流量预测[J]. 浙江大学学报(工学版), 2024, 58(7): 1366-1376.
[13]	何勇禧,韩虎,孔博. 基于多依赖图和知识融合的方面级情感分析模型[J]. 浙江大学学报(工学版), 2024, 58(4): 737-747.
[14]	李松,王哲,张丽平. SL-tgStore：新的时序知识图谱存储模型[J]. 浙江大学学报(工学版), 2024, 58(3): 449-458.
[15]	毛福新,杨旭,程嘉强,彭涛. 基于多模态融合的开放域三维模型检索算法[J]. 浙江大学学报(工学版), 2024, 58(1): 61-70.

Viewed

Full text

Abstract

Cited

Shared

Discussed