Abstract:

Visualization reconstruction technology aims to transform graphics into data forms that can be parsed and operated by machines， providing the necessary basic information for large-scale analysis， reuse and retrieval of visualization. However， the existing reconstruction methods focus on the recovery of visual information obviously， while ignoring the key role of interaction information in data analysis and understanding. To address the above problem， a visual interaction information reconstruction method for machine understanding was proposed. Firstly， interactions were defined formally to divide the visual elements into different visual groups， and the automated tools were used to extract interaction information of the visual graphics. Secondly， associations among interactions and visual elements were decoupled， and the interactions were split into independent experimental variables to build an interaction entity library. Thirdly， a standardized declarative language was formulated to realize querying of the interaction information. Finally， migration rules were designed to achieve migration adaptation of interactions among different visualizations based on visual element matching and adaptive adjustment mechanisms. The experimental cases focused on downstream tasks for machine understanding， such as visual question answering， querying， and migration. The results show that adding interaction information can enable machines to understand the semantics of visual interaction， thereby expanding the application scope of the above tasks. The above experimental results verify that proposed method can achieve structural integrity of the reconstructed visual graphics by integrating dynamic interaction information.

Key words: machine understanding, interaction, reconstruction of interaction information, visualization reconstruction, data analysis

摘要：

可视化重构技术旨在将图形转换为机器能够解析和操作的数据形式，为可视化的大规模分析、重用及检索等提供必备的基础信息；然而，现有的重构方法明显侧重于视觉信息的恢复，忽视了交互信息在数据分析和理解中发挥的关键作用。针对上述问题，提出一种面向机器理解的可视化交互信息重构方法。首先，形式化定义交互，将可视元素划分为不同的视觉组，采用自动化工具提取可视化图形的交互信息；其次，解耦交互与可视元素的关联，将交互分离为独立的实验变量，构建交互实体库；再次，制定规范的声明式语言，实现交互信息的查询；最后，设计迁移规则，基于可视元素匹配与自适应调整机制实现交互在不同可视化间的迁移适配。实验案例针对可视化问答、查询和迁移等面向机器理解的下游任务，结果显示增加交互信息能够使机器理解可视化交互的语义，从而拓展上述任务的应用范围。以上实验结果验证了所提方法能够使重构后的可视化图形通过融合动态交互信息而达成结构完整性。

关键词: 机器理解, 交互, 交互信息重构, 可视化重构, 数据分析