Reverse Engineering Stack Exchange is a question and answer site for researchers and developers who explore the principles of a system through analysis of its structure, function, and operation. It only takes a minute to sign up.
Sign up to join this community
I want to find a way to improve my unpacking skills, i am not a noob but i miss steps from simple to hard packers.
I saw lot of video tutorials, the most of the time i see "click F9 x times to reach the OEP" or "ESP trick" without go deeper as i want. I find the learning process slow in this way.
I am searching for paper or books to go deeper in this art, my goal is to be able to face malware analisis in the best way.
Please advice, thanks
There is not a single paper or book that explains the art of unpacking. This is mainly due to all the different packers that require different techniques to unpack them. There are of course generic approaches that work on some packers but knowing when to use them is gained from experience. IMHO the best approach to learn unpacking is to follow along with unpackmes in a debugger. My favorite unpacking author is Haggar. Once you go through a handful of the tutorials you will observe patterns of instructions and API sequences. Once you see those patterns you should try to unpack the sample using what you previously learned. If you fail, you can continue on with the tutorial and if you succeed, you can move on to the next tutorial. Either way you learn something new or apply previously gained knowledge. In the long run it will help you with unpacking malware.
Unpacking is an art, and as such there aren't clear steps to unpack any target.
I was in your situation a few years ago. I understood assembly, and code injections and hooking and whatnot, but I could not grasp unpacking.
The key is to approach unpacking analytically. You have a state A, which is a (possibly!) packed executable, and you have a desired state B which contains code you can analyze in IDA for example. You want to find where that state switch happens.
This sounds rather generic, and that's because it is. Especially in malware unpacking, it's not always 100% clear when code is unpacked. Sometimes there is no unpacked binary, but injected shellcode. Sometimes you get 3 stages of unpacking.
So then, this is not helpful. That's why you start with very simple approaches to learn the patterns.
That's where the ESP trick you mentioned comes into play. This is a trick that commonly allows you to skip code. Oftentimes, unpacking stubs are surrounded with pushad/popad to not taint the environment before the actual code runs. That's one pattern to learn - something starts with saving the execution state (more or less) - so we're interested in what happens afterwards.
Another idea: PE executables have a header. So if some unpacker unpacks the target code, does it restore the original header? The answer surprisingly is mostly yes! That's great, because now we can just watch for the header being changed and are close to the final form of the code. This can be done be setting a Hardware Breakpoint on Write on some field in the header - for example the AddressOfEntryPoint, because that's what we want to know!
Then there are so-called RunPE packers (also now referred to as 'process hollowing') - they broke up with the concept of unpacking as in a small stub that rebuilds the original executable in-placce but instead create a new (possibly unrelated) process, then rewrite that new process to the state of the original binary.
So how do we handle that? We watch for APIs that are used. We watch for CreateProcess(CREATE_SUSPENDED), we watch WriteProcessMemory, we watch ResumeThread. Of course, there are dozens of variations (ZwWriteVirtualMemory, ZwResumeProcess, ZwCreateProcess and so on) but the key steps are the same: Create some new process, modify it, resume execution.
Then there are examples that simply don't fit a scheme themselves - what to do with them? Follow their code flow! Look until you see something you can use. Sometimes you'll end up with some dumped shellcode from WriteProcessMemory, but that's the 'final' stage B, there is nothing else.
This is a long answer but my point is: Unpacking is simple - packed code is transformed into unpacked code. There is no magic between these steps, so follow the code and think outside the box.
Sometimes all you need is a breakpoint on VirtualFree() to catch the unpacked code being freed. Sometimes you just need a breakpoint on VirtualAlloc() and watch what is written there. By thinking about what CAN happen between state A and B you'll find points to 'ambush' the unpacked code so to speak.
This answer is mostly applicable for Linux as thats where I do most of my RE. I think you might be able to use it for windows too.
First, I'm assuming that no antidebugging or antiemulation tricks are applied. Here's how I would approach the problem. My ultimate aim would be to try and find the OEP. I'd try running the binary in a VM, and perhaps try to see what it does using strace/procmon. For eg:- perhaps it reads a value from a register or tries to see if a file exists. If so, I try to debug the binary, catch the syscall that reads from the registry/file and look at the process map. From here I try to deduce, what parts of the memory map belong to the unpacked process. At this point, I try to debug from the start of the program again, trying to reach the OEP. Generally it is a good idea to turn off ASLR when doing all this.
In case anti-debugging tricks are present, it goes without saying that a familiarity and knowledge of these tricks will give you an edge as you can spot them with relative ease.
Other RE tricks like changing the protection of a region to NX so that an access violation occurs when trying to execute it; or flooding a region with 0xcc's so that a SIGTRAP occurs when trying to execute it might also be useful.
Hope this helps.
