Abstract:

The video question answering task aims to understand video content deeply and answer natural language questions. Current methods based on Multimodal Large Language Models （MLLMs） struggle to model global semantics and temporal dependencies across scenes effectively due to dual constraints of context length and computational complexity， resulting in degraded model inference performance. To address this issue， a Scene-Driven Adaptive Keyframe Sampling （SD-AKS） method and a Question-Driven Video Summarization （QD-VS） method were proposed. First， an iterative video scene clustering based on K-means was introduced， and a Scene Separation Score （SSS） was designed， so as to evaluate feature differences between scenes quantitatively， thereby enhancing the accuracy and robustness of scene segmentation. Second， a problem semantic-guided adaptive keyframe sampling was designed to achieve dynamic keyframe selection based on quantified scene information density and fine coverage of information-dense regions. Furthermore， based on the keyframe method， Large Language Models （LLMs） were employed to summarize the video-related question set， so that the task-oriented textual summaries were generated to improve global model inference capability. Experimental results show that compared to LLaVA?Video， the best?performing baseline model， the proposed method improves prediction accuracy by 5.3 and 1.8 percentage points on the EgoSchema and NExT?QA datasets， respectively， which verifies its effectiveness in cross?dataset scenes.

Key words: video question answering, Multimodal Large Language Model (MLLM), keyframe localization, multimodal reasoning, video summarization

摘要：

视频问答任务旨在深度理解视频内容并回答自然语言问题。当前基于多模态大语言模型（MLLM）的方法在处理长视频时，受限于上下文长度和计算复杂度的双重限制，难以有效建模跨场景的全局语义与时序依赖，造成模型推理性能退化。针对此问题，提出基于场景驱动的自适应关键帧采样（SD-AKS）方法和问题驱动的视频摘要生成（QD-VS）方法。首先，引入基于K-means的迭代视频场景聚类，设计场景分离分数（SSS），量化评估场景间的特征差异，从而提高场景分割的精度和鲁棒性；其次，设计问题语义引导的自适应关键帧采样，实现基于场景信息密度量化的动态关键帧选择和对信息密集区域的精细覆盖。此外，在关键帧方法的基础上，利用大语言模型（LLM）总结视频关联问题集，生成任务导向的文本摘要，提升模型的全局推理能力。实验结果表明，所提方法在EgoSchema和NExT-QA数据集上的预测准确率与基线模型中表现最优的LLaVA-Video相比，分别提升了5.3与1.8个百分点，验证了它在跨数据集场景下的有效性。

关键词: 视频问答, 多模态大语言模型, 关键帧定位, 多模态推理, 视频摘要