You don't throw things at IDA and hope for the best and, as an analyst, you don't relay on automatic tools this much.
That's clearly a sign you need to start doing the heavy lifting.
A packer is simply a binary, so you proceed as you would do with any other binary.
There is no general procedure, I'll tell you what I usually do but I may have forgotten something and the list cannot possibly be exhaustive.
The general rule is: you debug it until you understand it, no matter how much time and focus will it take (5 min, 1 hour, 3 days, a month, doesn't matter).
You do this often enough and the time will shorten further and further until it's acceptable, you never get your hands dirty (i.e. use automated tools) and you'll never learn how to do this.
Anyway, I'd do this assuming the packer is a PE:
- Inspect the PE with a PE viewer. I use CFF Explorer. This will tell you if the PE is 32 or 64-bit, if it's a .NET assembly and what resources it has. Very simple packers have plain or XOR/RC4 encrypted payloads in their resources. The resources will also tell you if you are dealing with a Delphi binary, a particular runtime, an installer, UPX and so on. With experience, you can even tell malware and packer families aparts just by inspecting their PE.
- If it's a .NET use dnspy to inspect and debug it.
- If it's a Delphi use IDR to inspect and debug it.
- If it's a VB6 use VB decompiler.
- If it's AutoIT or similar scripted automation, use the relevant decompiler.
- If it's an installer (NSIS, MSI, SFX, ...) use 7z 9.38 to extract it and then go back to point 1 (you may find out the packer it's an installer later, it's OK, you are required to know how the main installer looks like when disassembled with IDA, Ghidra or similar).
- If it's not either use IDA.
- If necessary debug the packer (I use x64dbg).
These can be easy to unpack or can be hard (if they patch metadata or use delegate shadowing).
In either case, check any module constructor and variable initializer. Often the malicious code is hidden inside a legal application, you have to find it starting from the entry-point. This is tedious but not hard, it's easy to spot malicious code (you are required to be a programmer to be a malware analyst, so you surely know what good code looks like).
Often the packers use Control-Flow Obfuscation, this makes the code hard to read but not too hard to debug.
What these packers usually do is decode an assembly (a DLL) from the resource and load it with
Assembly.Load then use reflections (
Invoke) to invoke a method. Use the module view of dnspy to put breakpoints in dynamically loaded assembly.
By debugging and inspecting the local variable you can easily find the configuration and the payload. You'll know when you are in the final unpacking routine because there will be a lot of checks on the configuration (cfr. MaaS).
Keep in mind that there can be one, two, five, or even ten additional assemblies before the final unpacking.
If the packer uses "delegate shadowing" (i.e. call its method and runtime methods through delegates) you can inspect the target of the delegate in the local window to see if you are going to step into a runtime function.
De4dot was once useful for deobfuscating .NET assembly, not it's pretty useless but for dynamically executing string deobfuscation and renaming.
dnlib is a very useful library to manipulate IL code, you are required to know how to use it to write deobfuscator or simplify the analysis.
This is a standard Delphi executable, as a malware analyst you are required to be confident with the Delphi runtime (particularly string and array manipulation, Unit initialization and GUI events).
You may need to download additional "Knowledge Bases" to have IDR recognize the Delphi runtime.
Your first task is finding the malicious code. This can be in a unit initialization routine, in a timer in a form or in any form load/create event.
Use IDR to inspect the possible entry-point and look for anything suspicious, use the rename function copiously.
Look for the usual tricks (PEB accesses,
LoadLibrary and so on). Generally speaking, the packer will allocate memory to decode the payload and then map and execute it.
There is no way to give a general procedure. Note that you want the payload just decoded but not yet mapped (the payload is usually a PE) otherwise you have to patch it once dumped.
First identify the runtime used, for example, Go, Rust, C, C++ or so on. You are required to know how to tell the main programming languages apart (including Delphi, sometimes it can be wrongly detected).
If the languages have metadata (e.g. Go, Rust) then use the necessary IDA script to annotate the DB from these metadata (e.g. Go has AlphaGolang). If the scripts don't exist, write them. As a malware analyst, you are required to be confident in the vast majority of programming languages and platforms, including idapy.
See also if FLIRT or Lumina can help you (particularly for C/C++).
Once you have made IDA detect as much library code as possible, you need to find and inspect the entry-point.
Get a gist of the possible obfuscation used and if used learn the patterns, what they actually mean and how to mentally translate them.
Learn the shape of the decoy code to ignore (this can be done with several years of experience in programming, decoy code makes no sense so it really stands out as you have never seen it while debugging the legal programs you worked on) and look for the suspicious one (again: PEB access, allocation of memory, loading of modules).
The packer will generally decode the payload in allocated memory and then map and execute it.
The memory can be allocated with a variety of functions (
VirtualAlloc, on a writable section, ...).
See if the decoding code is simple (short actually) enough to be worth emulating in a script or if it's easier to debug it.
You can expect an intermediate shellcode, this can be called with anything (indirect
push/ret pairs, callbacks in APIs, ...).
Again, you want the payload decoded but not mapped.
If the packer is really written in VB6, just read the decompiled code. Again it should be easy to spot the decoding routine thanks to the boatload of metadata.
Often the VB6 binary is just a decoy, you have to find the call to the shellcode. This is usually a simple
call to a fixed address that is not listed as a VB6 routine. Put a breakpoint there and debug.
You are also required to know the anti-debug tricks and how to work around them (you can simply put breakpoint on APIs).