How to unpack files packed with multiple packers?

Question

As I am just getting started in RE, I've mostly faced files packed with a single-layer of packing , such as UPX, ASPack, etc.

Unpacking these protections is fully documented online. The problem begins when I deal with multiple layers of packing, especially concerning malware. I have followed some tutorials though they're usually not detailed enough. They seem to go through a tedious process to find the OEP. For example, they start by dealing with common packers (which is the easy part) and then they begin to set breakpoints everywhere "in calls and jumps" and tracing through the file here and there, which is for me the hard part that I have described above. At this point, I have no clue for what they are seeking or for what they are aiming, and then after some work, they find the OEP!

So what logic did they follow in that process? Also, because I know that the subject is broad, I'm also interested in some keywords.

Answer 1

As you saw by yourself, RE is not a easy skill to accomplish. One must find as much patience (at least) as he could to acquire it, some intelligence and knowledge will also help. If you think that just following the tutorial you will understand all of it from the first read, then you are wrong.
My advice to you is :

• read the tutorial several times and try to pass it without looking into the writings
• try to understand as much as you can about OS/platform you are reversing on
• be proficient with PE format for a start or the file format you are dealing with
• read about packers and obfuscation techniques

After that, I'm pretty sure, if you come back to that tut again, you will understand it more clearly.

Here are some good resources:

1. Malware Unpacking In Ollydbg
2. Finding OEP and unpacking Malware packed with ASPACK 2.12 - try it although you said, you came across similar ones
3. Fravia's archive pages of reverse engineering
4. Overview of PE file format

May the force be with you :)

Answer 2

To Unpack a file you must have quite a lot of experience in reversing binaries..

Remember there exists no universal method that works for all the packers.

These steps will work in unpacking 97% of binaries;

1. You must be aware of the code that usually lies at the start of entry point in binaries compiled by well known compilers VC++, VB, Borland Delphi and other compilers. You should also be aware of the difference in code near entry point in binaries compiled in different compiler versions. This will eventually help you in finding the OEP.
2. You must have abundance patience to follow all the virtual memory allocation and virtual memory freeing.
3. Dump the memory blocks and look for visible strings after execution of a decryption code.
4. Learn about debugger detection, virtual environment detection and anti-debugging techniques.
5. Start with simple packers at first. I will recommend you to try UPX older versions.
6. Last but not the least "Play with your favorite debugger at-least for 14 hours a day."

All the best and Happy reversing!!!