Suppose you want to conceal an application's true purpose from analysts by reacting to dynamic analysis tools. How is it possible to detect running (in debugger mode) the application under a reverse engineering tool (e.g., IDA Pro) by the application itself? What code do you propose to use while preparing the application?
the easy answer:
probably a Sleep(random()*60000 + 10000) is more than enough to thwat most dynamic analysis systems. Or if((int)(random()*10) == 8) and so on
check for debuggers:
The 'standard' way is to use IsDebuggerPresent. But analysis systems are aware of that and try to patch these calls. So there are similar approaches checking the field in the EPROCESS structure directly. I've also seen malware checking the result of OutputDebugString.
check for analysis systems
Some malware families employ checks for certain process names implying the system may be used for analysis, like the pcap driver or the names of debugger-processes. Also, the availability of certain libraries may prevent the program from exhibiting its real behavior.
check for virtual environment
There are already lots of papers about this. Most dynamic analysis of potentially malicious programs happens in virtual environments. There are different approaches you can take here:
- Timing traps - virtualization is slower than bare metal machines
- Undefined behavior - often virtualization tools respond differently in error situations (e.g. invalid opcodes)
- Suspicious strings - e.g. 'VirtualBox Harddrive', ...
Also, there are some very advanced approaches, like checking the behavior of the TLB.