Anti-disassembler techniques and IDA Pro

Question

I am a newbie to RE and IDA Pro 7.2, and I am working my way through the book Practical Malware Analysis. I'm currently working on the lab 15-01 problem in the anti-disassembler chapter, but am having difficulty reproducing part of the solution in my environment.

I solved the lab by redefining some bogus opcodes into data, and now I have reasonable disassembly listings to look at. So far, so good. But at the end, the solution tells you to turn this fixed-up section into a function using the "P" hotkey. That doesn't work for me: I get this message in the output window:

.text:00401010: The function has undefined instruction/data at the specified address.
Your request has been put in the autoanalysis queue.

The address points to the first data definition I made.

Trying to flip over to graph mode doesn't work either. I get a popup that tells me that only instructions belonging to functions can be displayed in graph mode.

If I go to the Problems tab, I see lines like this:

.text:00401062  NONAME      db 0E8h

But I didn't glean anything in the IDA Pro help file for no name problems that helped me with my specific situation.

Ultimately, I used Edit -> Patch Program -> change byte to NOP out the data bytes and turn them into code. Then I had a function that I could graph.

My question is, is there a way to do this without patching bytes?

EDIT: Added error message text to the post.

Answer 1

I think you may face what's called a rogue-byte.

I'm currently scripting something to patch them (see IDAPython - Find the start of an undefined code blob).

Can you tell me if it's the same type of code logic?

If yes, you indeed might want to replace the rogue-byte with a NOP. You can't do it without patching the binary, because as you said, it's an anti-disassembly technique. It's here in order to mess with IDA Pro!

If you want to patch it:

• select the line containing the rogue-byte, then undefine it (shortcut: "U", or go to "Edit -> Undefine")
• select the instruction ("db 0E8h") then go to "Edit -> Patch program -> Assemble". You can now replace your rogue-byte by a NOP, by typing "nop", then "OK".
• You now need to define the NOP as code (so it can get interpreted as an instruction), by pressing the C shortcut, or by going to "Edit -> Code".
• Finally, you can define your function as a proper function, by reaching the start of it (loc_XXXXX for instance) and pressing the P shortcut, or "Edit -> Function -> Create function".

If you want, I can share you the IDAPython script that I wrote, even if it's still not complete yet.