Isolating encrypt/decrypt instructions in an execution trace

Question

If you have an execution trace of a program, and know it uses say AES for encryption.

Can you isolate the instructions for encryption with that knowledge alone?

Answer 1

Generally speaking, it depends on the platform. For example, if the software uses Intel AES extensions it is possible to find the corresponding instructions in the disassembly. If the software is compiled for other platforms and uses specific hardware accelerators it is possible to find it by accesses to specific addresses of the accelerators registers.

If there is no specific accelerator or specific instruction set it is possible to find S-BOX constants and check which instructions are accessing it.

There are some plugins for IDA that able to do this work for you, for example FindCrypt2.

In addition there is a presentation from recon conference about locating such an algorithms in obfuscated code here.