Identification of encrypted function in malicious software