thank for reading, i'm a beginner trying to learn Malware Analysis and Reverse Engineering, at the moment I readed the book "Learning Malware Analysis", I fully understand it, but there is just one section that i can't grasp, Unpacking, I tried to manually unpack UPX without success even if I watched tutorial and did the exact same thing (Breakpoint on the famous jump and then dump with scylla and rebuild IAT, but nothing). I don't understand how detect the instructions that unpack the executable, can someone give me the right resources to start learning unpacking? I already have tutorials like "unpacking xxxx" but I would prefere something more general, concepts that I need to know, I hope you understand my question, tell me if I said something wrong, thank you.
2 Answers
Manually unpacking is an art that takes lots of practice, but you're starting off right by working on the most simple examples. You need to have a solid understanding of the tools involved in the process and the PE file format. There are quite a few resources out there, but I've had great success spinning up folks with [yates] old docs on PE files, and Lena's tutorials (the unpacking specific things) Lena's tuts are old, but for understanding some core concepts they're still pretty relevant (x86 isn't completely dead just yet). If you want to go forward with more, there are quite a few unpackme's out for different targets too. You should be able to follow along with relative ease using x64dbg instead of olly for those tutorials as well.
So for starters, I'd suggest snagging Lena's tuts. If you're use to the toolset you probably can jump right in with Part 20 & 21, part 21 has a UPX unpackme, but part 20 teaches some other core concepts.
Familiarize yourself with the PE file format, I really liked [yates] info on PE files also his Understanding Import tables article is also a great read. Finally, his Program Flow paper ties a lot of that info together well, with some really solid and simple x86/win32 asm code examples to show you how to write your own tools.
Also a decent resource on PE things is Goppit's Code Breaker article. It's rather in depth, but covers about any question you could have.
That should be enough to get you started down the right path, knowing your tools and knowing what docs to read is definitely half the battle. Good luck, you can do it. archive.org and is your friend if you can't find some of the files linked in those tuts.
learn the basics of how packers works; I haven't read the book that you mentioned, but it should discuss it;
Detect whether it uses custom packer or known (commercial) packer:
- there is known sections; UPX0
- tools like Detect It Easy will identify it;
- if that was not the case, it's mostly custom packer;
Unpacking known packers:
- usually there is pushad/popad combination that lead to OEP.
- there is far jump; meaning that sample will jump from one section to another, or similar thing. jump target usually has no actual meaning unless sample reach that address. (actual instructions will be written there). push "address" ret; jump "register"; jump "address"
- you will duckduckgo it and find some forums that talk about this particular packer.
if these are the case, follow: https://www.virusbulletin.com/virusbulletin/2012/04/quick-reference-manual-unpacking
Unpacking Custom Packers:
- trace it's API Calls either with DBI or inline hooking; there are tools out there that do it for you. find the point that the malware starts it's actual malicious activity; put breakpoint on corresponding APIs and dump the payload. I usually put breakpoint on VirtualAlloc, VirtualProtect, CreateProcessInternalW, WriteProcessMemory depending on sample.
