Improved instruction obfuscation framework based on obfuscator low level virtual machine

doi:10.11772/j.issn.1001-9081.2021122234

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (2): 490-498.DOI: 10.11772/j.issn.1001-9081.2021122234

• Computer software technology • Previous Articles

Improved instruction obfuscation framework based on obfuscator low level virtual machine

Yayi WANG, Chen LIU, Tianbo HUANG, Weiping WEN()

School of Software and Microelectronics，Peking University，Beijing 102600，China

Received:2022-01-06 Revised:2022-05-20 Accepted:2022-05-24 Online:2022-06-13 Published:2023-02-10
Contact: Weiping WEN
About author:WANG Yayi， born in 1998， M. S. candidate. Her research interests include code obfuscation， vulnerability mining.
LIU Chen， born in 1999， M. S. candidate. His research interests include code obfuscation， vulnerability mining.
HUANG Tianbo， born in 1997， M. S. candidate. His research interests include cyberspace security， malicious code detection， code obfuscation.
Supported by:
Peking University Horizontal Research Project(2020001763)

改进的基于底层虚拟机混淆器的指令混淆框架

王雅仪, 刘琛, 黄天波, 文伟平()

北京大学软件与微电子学院，北京 102600

通讯作者: 文伟平
作者简介:王雅仪（1998—），女，四川成都人，硕士研究生，主要研究方向：代码混淆、漏洞挖掘
刘琛（1999—），男，河南新乡人，硕士研究生，主要研究方向：代码混淆、漏洞挖掘
黄天波（1997—），男，河北邯郸人，硕士研究生，主要研究方向：网络空间安全、恶意代码检测、代码混淆；
基金资助:
北京大学横向课题(2020001763)

Abstract

Abstract:

Focusing on the issue that only one instruction substitution with 5 operators and 13 substitution schemes is supported in Obfuscator Low Level Virtual Machine （OLLVM） at the instruction obfuscation level， an improved instruction obfuscation framework InsObf was proposed. InsObf， including junk code insertion and instruction substitution， was able to enhance the obfuscation effect at the instruction level based on OLLVM. For junk code insertion， firstly， the dependency of the instruction inside the basic block was analyzed， and then two kinds of junk code， multiple jump and bogus loop， were inserted to disrupt the structure of the basic block. For instruction substitution， based on OLLVM， it was expanded to 13 operators， with 52 instruction substitution schemes. The framework prototype was implemented on Low Level Virtual Machine （LLVM）. Experimental results show that compared to OLLVM， InsObf has the cyclomatic complexity and resilience increased by almost four times， with a time cost of about 10 percentage points and a space cost of about 20 percentage points higher. Moreover， InsObf can provide higher code complexity compared to Armariris and Hikari， which are also improved on the basis of OLLVM， at the same order of magnitude of time and space costs. Therefore， InsObf can provide effective protection at the instruction level.

Key words: software protection, code obfuscation, instruction obfuscation, Obfuscator Low Level Virtual Machine (OLLVM), junk code insertion, instruction substitution

摘要：

针对底层虚拟机混淆器（OLLVM）在指令混淆层面只支持指令替换一种算法，且仅支持5种运算符和13种替换方案的问题，设计了一种改进版的指令混淆框架InsObf，以加强OLLVM指令层面的混淆效果。InsObf包含指令加花和指令替换，其中指令加花首先对基本块的指令进行依赖分析，然后插入叠加跳转和虚假循环两种花指令；指令替换在OLLVM的基础上，拓展至13种运算符，共计52种指令替换方案。在底层虚拟机（LLVM）上实现了框架原型后，通过实验表明，与OLLVM相比，InsObf在时间开销增长约10个百分点，空间开销增长约20个百分点的情况下，圈复杂度和抗逆向能力均可提高近4倍；与同样基于OLLVM改进的Armariris和Hikari相比，InsObf在同一量级的时空开销下，可以提供更高的代码复杂度。因此，InsObf可提供指令层级的有效保护。

关键词: 软件保护, 代码混淆, 指令混淆, 底层虚拟机混淆器, 指令加花, 指令替换

CLC Number:

TP311

Yayi WANG, Chen LIU, Tianbo HUANG, Weiping WEN. Improved instruction obfuscation framework based on obfuscator low level virtual machine[J]. Journal of Computer Applications, 2023, 43(2): 490-498.

王雅仪, 刘琛, 黄天波, 文伟平. 改进的基于底层虚拟机混淆器的指令混淆框架[J]. 《计算机应用》唯一官方网站, 2023, 43(2): 490-498.

Figures/Tables 13

References 46

1	COLLBERG C， THOMBORSON C， LOW D. A taxonomy of obfuscating transformations： technical reports 148［R］. Auckland： The University of Auckland， 1997.
2	KUANG K Y， TANG Z Y， GONG X Q， et al. Enhance virtual-machine-based code obfuscation security through dynamic bytecode scheduling ［J］. Computers and Security， 2018， 74： 202-220. 10.1016/j.cose.2018.01.008
3	ZHAO Y J， TANG Z Y， YE G X， et al. Compile-time code virtualization for android applications ［J］. Computers and Security， 2020， 94： No.101821. 10.1016/j.cose.2020.101821
4	ZHANG X S， HE F L， ZUO W L. An inter-classes obfuscation method for Java program［C］// Proceedings of 2008 International Conference on Information Security and Assurance. Piscataway： IEEE， 2008： 360-365. 10.1109/isa.2008.49
5	SHOSHITAISHVILI Y， WANG R Y， SALLS C， et al. SOK：（state of） the art of war： offensive techniques in binary analysis［C］// Proceedings of 2016 IEEE Symposium on Security and Privacy. Piscataway： IEEE， 2016： 138-157. 10.1109/sp.2016.17
6	MU D L， GUO J， DING W B， et al. ROPOB： obfuscating binary code via return oriented programming［C］// Proceedings of 2017 International Conference on Security and Privacy in Communication Systems， LNICST 238. Cham： Springer， 2018： 721-737. 10.1007/978-3-319-78813-5_38
7	JUNOD P， RINALDINI J， WEHRLI J， et al. Obfuscator-LLVM — software protection for the masses［C］// Processing of IEEE/ACM 1st International Workshop on Software Protection. Piscataway： IEEE， 2015： 3-9. 10.1109/spro.2015.10
8	曹林，孙国梓，王海平，等. 花指令模糊变换逻辑一致性研究［J］. 计算机工程， 2006， 32（20）： 135-137， 152. 10.3969/j.issn.1000-3428.2006.20.049
	CAO L， SUN G Z， WANG H P， et al. Logic invariability study of junk code transformation［J］. Computer Engineering， 2006， 32（20）： 135-137， 152. 10.3969/j.issn.1000-3428.2006.20.049
9	杨宇波，黄玮，李忠献，等. 抵御静态反汇编的代码混淆算法［J］. 北京理工大学学报， 2015， 35（3）： 272-278.
	YANG Y B， HUANG W， LI Z X， et al. Code obfuscation algorithm to resist static disassembly［J］. Transactions of Beijing Institute of Technology， 2015， 35（3）： 272-278.
10	孙国梓，陈丹伟，蔡强. 子程序花指令模糊变换逻辑一致性研究［J］.计算机科学， 2009， 36（8）： 89-91， 200. 10.3969/j.issn.1002-137X.2009.08.022
	SUN G Z， CHEN D W， CAI Q. Research on logic consistency of junk code transformation within sub-function［J］. Computer Science， 2009， 36（8）： 89-91， 200. 10.3969/j.issn.1002-137X.2009.08.022
11	BANESCU S， COLLBERG C S， GANESH V， et al. Code obfuscation against symbolic execution attacks［C］// Proceedings of the 32nd Annual Conference on Computer Security Applications. New York： ACM， 2016： 189-200. 10.1145/2991079.2991114
12	HALEVI S， HALEVI T， SHOUP V， et al. Implementing BP-obfuscation using graph-induced encoding［C］// Proceedings of 2017 ACM SIGSAC Conference on Computer and Communications Security. New York： ACM， 2017： 783-798. 10.1145/3133956.3133976
13	ISMANTO R N， SALMAN M. Improving security level through obfuscation technique for source code protection using AES algorithm［C］// Proceedings of the 7th International Conference on Communication and Network Security. New York： ACM， 2017： 18-22. 10.1145/3163058.3163071
14	潘雁，祝跃飞，林伟. 基于指令交换的代码混淆方法［J］. 软件学报， 2019， 30（6）： 1778-1792.
	PAN Y， ZHU Y F， LIN W. Code obfuscation based on instructions swapping［J］. Journal of Software， 2019， 30（6）： 1778-1792.
15	CECCATO M， TONELLA P. CodeBender： remote software protection using orthogonal replacement［J］. IEEE Software， 2011， 28（2）： 28-34. 10.1109/ms.2010.158
16	ARMOOGUM S， CAULLY A. Obfuscation techniques for mobile agent code confidentiality［J］. Journal of Information and Systems Management， 2011， 1（1）： 25-36.
17	CHO S， CHANG H， CHO Y. Implementation of an obfuscation tool for C/C++ source code protection on the XScale architecture［C］// Proceedings of 2008 IFIP International Workshop on Software Technologies for Embedded and Ubiquitous Systems， LNCS 5287. Berlin： Springer， 2008： 406-416.
18	BALACHANDRAN V， EMMANUEL S， KEONG N W， et al. Obfuscation by code fragmentation to evade reverse engineering［C］// Proceedings of the 2014 IEEE International Conference on Systems， Man， and Cybernetics. Piscataway： IEEE， 2014： 463-469. 10.1109/smc.2014.6973951
19	ZHANG Y J， PANG J M. A new compile-time obfuscation scheme for software protection［C］// Proceedings of 2016 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery. Piscataway： IEEE， 2016： 1-5. 10.1109/cyberc.2016.10
20	COHEN F B. Operating system protection through program evolution［J］. Computers and Security， 1993， 12（6）： 565-584. 10.1016/0167-4048(93)90054-9
21	PENG Y， SU G Y， TIAN B， et al. Control flow obfuscation based protection method for Android applications［J］. China Communications， 2017， 14（11）： 247-259. 10.1109/cc.2017.8233664
22	JACKSON T， HOMESCU A， CRANE S， et al. Diversifying the software stack using randomized NOP insertion［C］// JAJODIA S， GHOSH A K， SUBRAHMANIAN， V S， et al. Moving Target Defense II： Application of Game Theory and Adversarial Modeling， ADIS 100. New York： Springer， 2013： 151-173. 10.1007/978-1-4614-5416-8_8
23	JANGDA A， MISHRA M， DE SUTTER B. Adaptive just-in-time code diversification［C］// Proceedings of the 2nd ACM Workshop on Moving Target Defense. New York： ACM， 2015： 49-53. 10.1145/2808475.2808487
24	LU K J， XIONG S Y， GAO D B. RopSteg： program steganography with return oriented programming［C］// Proceedings of the 4th ACM Conference on Data and Application Security and Privacy. New York： ACM， 2014： 265-272. 10.1145/2557547.2557572
25	LINN C， DEBRAY S. Obfuscation of executable code to improve resistance to static disassembly［C］// Proceedings of the 10th ACM Conference on Computer and Communications Security. New York： ACM， 2003： 290-299. 10.1145/948109.948149
26	乐德广，赵杰，龚声蓉. 基于模式切换的ARM汇编代码混淆算法［J］.计算机工程与应用， 2021， 57（18）： 122-129.
	LE D G， ZHAO J， GONG S R. ARM assembly code obfuscation algorithm based on mode switch［J］. Computer Engineering and Applications， 2021， 57（18）： 122-129.
27	WU J H， GAO X C， TIAN C H， et al. The study for protecting mobile agents based on time checking technology［C］// Proceedings of 2007 IEEE International Conference on Robotics and Biomimetics， Piscataway： IEEE， 2007： 2013-2017. 10.1109/robio.2007.4522476
28	RAJBA P， MAZURCZYK W. Data hiding using code obfuscation［C］// Proceedings of the 16th International Conference on Availability， Reliability and Security. New York： ACM， 2021： No.75. 10.1145/3465481.3470086
29	SU Q， WANG Z Y， WU W M， et al. Technique of source code obfuscation based on data flow and control flow transformations［C］// Proceedings of 7th International Conference on Computer Science and Education. Piscataway： IEEE， 2012： 1093-1097. 10.1109/iccse.2012.6295254
30	PICHETA D. Code obfuscation for the C/C++ language［D/OL］. Belfast： Queen’s University of Belfast. （2018-05-02）［2021-12-29］..
31	KANZAKI Y， MONDEN A， NAKAMURA M， et al. Exploiting self-modification mechanism for program protection［C］// Proceedings of 27th Annual International Computer Software and Applications Conference. Piscataway： IEEE， 2003： 170-179. 10.1109/cmpsac.2003.1245313
32	KANZAKI Y， MONDEN A， NAKAMURA M， et al. A software protection method based on instruction camouflage［J］. Electronics and Communications in Japan （Part III： Fundamental Electronic Science）， 2006， 89（1）： 47-59. 10.1002/ecjc.20141
33	DARWISH S M， GUIRGUIS S K， ZALAT M S. Stealthy code obfuscation technique for software security［C］// Proceedings of the 2010 International Conference on Computer Engineering and Systems. Piscataway： IEEE， 2010： 93-99. 10.1109/icces.2010.5674830
34	LATTNER C， ADVE V. LLVM： a compilation framework for lifelong program analysis & transformation［C］// Proceedings of the 2004 International Symposium on Code Generation and Optimization. Piscataway： IEEE， 2004： 75-86. 10.1109/cgo.2004.1281650
35	GoSSIP-SJTU/Armariris［EB/OL］. （2019-06-09）［2021-12-29］.. 10.12677/ces.2021.93120
36	Obfuscator Hikari. Hikari： LLVM obfuscator［EB/OL］. （2020-01-30）［2021-12-29］..
37	Clang： a C language family frontend for LLVM［EB/OL］. （2021-10-04）［2021-12-29］.. 10.1109/llvm-hpc.2018.8639205
38	CHEN C F， PETSIOS T， POMONIS M. CONFUSE： LLVM-based code obfuscation［EB/OL］. （2013-01-30）［2021-05-16］..
39	PALSBERG J， KRISHNASWAMY S， KWON M， et al. Experience with software watermarking［C］// Proceedings of the 16th Annual Computer Security Applications Conference. Piscataway： IEEE， 2000： 308-316.
40	GeeksforGeeks. A computer science portal for geeks［EB/OL］. ［2021-12-29］..
41	McCABE T J. A complexity measure［J］. IEEE Transactions on Software Engineering， 1976， SE-2（4）： 308-320. 10.1109/tse.1976.233837
42	JuliaComputingOSS. LLVM-CBE： resurrected LLVM "C Backend"， with improvements［EB/OL］. （2021-07-19）［2021-12-29］..
43	YIN T. Lizard： a simple code complexity analyser without caring about the C/C++ header files or Java imports， supports most of the popular languages［EB/OL］. （2021-12-23）［2021-12-29］..
44	Rays Hex. IDA Pro［EB/OL］. ［2021-12-29］..
45	eShard. D810［EB/OL］. （2021-10-08）［2021-12-29］..
46	Zynamics. BinDiff［EB/OL］. ［2021-12-29］..

运算符	等价指令序列
a=b+c	a = b-（-c）
	a = -（-b+（-c））
	r = rand（）， a = b+r， a = a+c， a=a-r
	r = rand（）， a = b-r， a = a+c， a=a+r
a=b-c	a = b+（-c）
	r = rand（）， a = b+r， a = a-c， a = a-r
	r = rand（）， a = b-r， a = a-c， a = a+r
a=b&c	a = （b^！c）&b
a=b&&c	a = ！（！b\|！c）&&（r\|！r）
a=b\|c	a = （b&c）\|（b^c）
a=b\|\|c	a = ［（！b&&r）\|\|（c&&！r）^（！c&&r）\|\| （c&&！r）］\|\|［！（！b\|\|！c）&&（r\|\|！r）］
a=b^c	a = （！b&c）\|（b&！c）
a=b^c	a = （！b&&r\|\|b&&！r）^（！c&&r\|\|c&&！r）

运算符	等价指令序列
a=b+c	a = b-（-c）
	a = -（-b+（-c））
	r = rand（）， a = b+r， a = a+c， a=a-r
	r = rand（）， a = b-r， a = a+c， a=a+r
a=b-c	a = b+（-c）
	r = rand（）， a = b+r， a = a-c， a = a-r
	r = rand（）， a = b-r， a = a-c， a = a+r
a=b&c	a = （b^！c）&b
a=b&&c	a = ！（！b\|！c）&&（r\|！r）
a=b\|c	a = （b&c）\|（b^c）
a=b\|\|c	a = ［（！b&&r）\|\|（c&&！r）^（！c&&r）\|\| （c&&！r）］\|\|［！（！b\|\|！c）&&（r\|\|！r）］
a=b^c	a = （！b&c）\|（b&！c）
a=b^c	a = （！b&&r\|\|b&&！r）^（！c&&r\|\|c&&！r）

运算符	OLLVM支持的替换方案数	InsObf支持的替换方案数
ADD	4	5
FADD	不支持	4
SUB	3	5
FSUB	不支持	4
AND	2	5
OR	2	5
XOR	2	5
MUL	不支持	3
FMUL	不支持	3
REM	不支持	3
SHL	不支持	4
LSHR	不支持	3
ASHR	不支持	3

运算符	OLLVM支持的替换方案数	InsObf支持的替换方案数
ADD	4	5
FADD	不支持	4
SUB	3	5
FSUB	不支持	4
AND	2	5
OR	2	5
XOR	2	5
MUL	不支持	3
FMUL	不支持	3
REM	不支持	3
SHL	不支持	4
LSHR	不支持	3
ASHR	不支持	3

混淆方法	圈复杂度增长率	混淆方法	圈复杂度增长率
ollvm-sub	128.8	InsObf-junk	300.6
InsObf-sub	231.7	InsObf-mix	607.2

Improved instruction obfuscation framework based on obfuscator low level virtual machine

改进的基于底层虚拟机混淆器的指令混淆框架

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 46

Related Articles 9

Recommended Articles

Metrics

混淆方法	时间开销增长率	空间开销增长率
ollvm-sub	21.6	20.6
InsObf-sub	23.3	25.1
InsObf-junk	15.0	36.4
InsObf-mix	31.0	42.5

混淆方法	开销增长率		圈复杂度增长率
混淆方法	时间	空间	圈复杂度增长率
InsObf-mix	31.0	42.5	607.2
Armariris-string	18.4	0.0	102.8
Hikari-string	36.1	23.8	215.0
Hikari-cf	40.1	36.6	—

[1]	Dajiang TIAN, Chengyang LI, Tianbo HUANG, Weiping WEN. Identifier obfuscation method based on low level virtual machine [J]. Journal of Computer Applications, 2022, 42(8): 2540-2547.
[2]	XIAO Shuntao, ZHOU Anmin, LIU Liang, JIA Peng, LIU Luping. Obfuscator low level virtual machine deobfuscation framework based on symbolic execution [J]. Journal of Computer Applications, 2018, 38(6): 1745-1750.
[3]	WANG Yan, HUANG Zhangjin, GU Naijie. Obfuscating algorithm based on congruence equation and improved flat control flow [J]. Journal of Computer Applications, 2017, 37(6): 1803-1807.
[4]	FANG Dingyi, DANG Shufan, WANG Huaijun, DONG Hao, ZHANG Fan. JavaScript code protection method based on temporal diversity [J]. Journal of Computer Applications, 2015, 35(1): 72-76.
[5]	WANG Rui YANG Qiuxiang CHEN Gouxi MA Qiaomei. Software protection game model based on divided-storage strategy [J]. Journal of Computer Applications, 2013, 33(09): 2525-2528.
[6]	JIANG Zifeng ZENG Guangyu WANG Wei GAO Hongbo. Research on implementation mechanism and detection technique of BIOS trapdoor [J]. Journal of Computer Applications, 2013, 33(02): 455-459.
[7]	Qin-gui XU Gui-xiong LIU Fu-rong GAO. Software protection model for measurement applications [J]. Journal of Computer Applications, 2011, 31(04): 970-974.
[8]	Li-Guo Zhou . Software protection technology based on selfsealing code block [J]. Journal of Computer Applications, 2009, 29(3): 817-822.
[9]	HUANG Jun,XU Juan,ZUO Hong-fu. Register code software encryption and protection based on RSA algorithm [J]. Journal of Computer Applications, 2005, 25(09): 2080-2082.