If you have an execution trace of a program, and know it uses say AES for encryption.
Can you isolate the instructions for encryption with that knowledge alone?
Reverse Engineering Stack Exchange is a question and answer site for researchers and developers who explore the principles of a system through analysis of its structure, function, and operation. It only takes a minute to sign up.
Sign up to join this communityGenerally speaking, it depends on the platform. For example, if the software uses Intel AES extensions it is possible to find the corresponding instructions in the disassembly. If the software is compiled for other platforms and uses specific hardware accelerators it is possible to find it by accesses to specific addresses of the accelerators registers.
If there is no specific accelerator or specific instruction set it is possible to find S-BOX constants and check which instructions are accessing it.
There are some plugins for IDA that able to do this work for you, for example FindCrypt2.
In addition there is a presentation from recon conference about locating such an algorithms in obfuscated code here.