关键词: 加密函数识别, 循环的输入和输出参数, 循环数据流图, 循环输入输出集合, 动态二进制插桩

Abstract: To resolve that the malware (malicious software) usually avoids security detection and flow analysis through encryption function, this paper proposed a scheme which can identify the encrypted function in malware. The scheme generated the dynamic loop data flow graph by identifying the loop and input/output of the loop in the dynamic trace. Then the sets of input/output were abstracted according to the loop data flow graph, the reference of known encrypted function was designed and the reference whose parameters were elements of the input sets was computed. If the result could match any element of the output sets, then the scheme could conclude the malware encrypts information by the known encrypted function. The experimental results prove that the proposed scheme can analyze the encrypted function of payload in the obfuscated malware.

Key words: encrypted function identification, input/output parameter of loop, loop data flow graph, input/output set in loop, dynamic binary instrument