关键词: 视觉Transformer, 大语言模型, 后门攻击, 数据投毒, 指令微调

Abstract: With the advancement of multimodal technology, the security issues of Visual Language Models (VLMs) have garnered significant attention. Current backdoor attack methods against VLMs suffer from low stealthiness, monotonous patterns, poor attack effectiveness, and high required costs. To address these issues, a patch-adaptive perturbation-based backdoor attack method was proposed for the first time from the perspective of the model's visual encoder architecture. Firstly, the attack target identifier was defined as a character string and encoded into an invisible perturbation signal using steganographic techniques. Secondly, building upon the patch encoding concept of the Vision Transformer (ViT) architecture, the perturbation was adaptively embedded into various image patches, making the poisoned images carrying triggers visually indistinguishable from clean ones. Finally, by poisoning the data during the instruction fine-tuning stage, the multimodal feature fusion process of the model was manipulated to hijack its cross-modal alignment, ultimately achieving the attack effect of inducing the model to output malicious text. Experimental results demonstrate that the proposed method achieves an attack success rate of 96% under a low poisoning rate not exceeding 5%, with a detection success rate remaining below 15%. Its stealthiness surpasses that of various advanced methods, and it demonstrates strong transferability across different VLMs.

Key words: Vision Transformer(ViT), Large Language Model(LLM), backdoor attacks, data poisoning, instruction fine-tuning