4

I'm trying to write my own disassembler for PE,PE+ and ELF executables but I'm stuck with a big problem on PE and PE+ executables.

I'm checking my work by comparing my output with objdump, and I found some (bad) opcodes appear in the disassembled program. I immediately checked the instruction manual to control these values; they are shown as invalid in instruction manual. Some examples:

Example from PE files:

40dad1:       d6                      (bad)

Some other:

402f1c:       ff                      (bad)
402f1d:       ff                      (bad)
402f1e:       ff                      (bad)
402f1f:       ff 01                   incl   (%ecx) #at last a valid instruction

These are valid when I check the manual, but I cannot understand this (it's a PE+ file, architecture is AMD64):

f0 db a5 4e 9c 95 68    lock (bad) [rbp+0x68959c4e]

f0 is lock prefix
db means its a x87 instruction
a5 is ModRM byte(10 100 101) and by looking mod and reg fields we can say it's an invalid instruction
4e 9c 95 68 is used as 4byte disp but why ?

Do we assume that it's an invalid indirect x87 opcode and we continue to read as it's a valid opcode? I suppose objdump chooses this path.

And what are these (bad) instructions for? It's clear that they are not for aligning; or am I doing something wrong?

Btw, I'm trying to disassemble my old projects and FireFox to check if my program works. I'm using objdump -z -d -M intel XXYYZZ.exe to disassemble.

2

You are correct, and objdump is actually wrong to disassemble such instruction, only to mark it as (bad) in a later stage.

Here is how it works:

db a5 4e 9c 95 68    lock (bad) [rbp+0x68959c4e]

db is one of the 87 FPU extensions, and a5 indeed is the Mod/rm byte. Now, for the FPU extensions, the rm part is "as usual" for all other mod/rm instructions, but the mod part indicates which instruction to use, from this small table:

DB /0     FILD mem4i
DB /2     FIST mem4i
DB /3     FISTP mem4i
DB /5     FLD mem10r
DB /7     FSTP mem10r

where the /x number indicates what got encoded in the mod part of the mod/rm byte (for completeness: db eX also forms some valid opcodes).

So of all available codes, those with the mod/rm patterns of ..001..., ..100.. and ..110... form 'bad' codes - but you only know this after parsing the mod/rm byte and checking this specific table for the opcode DB.

Now apparently objdump parses the entire instruction - including the 4-byte immediate value - before checking if the base instruction is valid to start with. I suppose it's just a table that says

"fild", "(bad)", "fist", "fistp", "(bad)", "fld", "(bad)", "fstp"

and the (bad) entries get used as if they are actually valid.

One could argue that it's immaterial, because both ways lead to the conclusion the opcode is 'bad', but with objdump's method, you are not only discarding the first byte as 'bad' but an entire 6 bytes. It is at least theoretically possible the first byte (which causes the entire next sequence to be invalid) is data but followed immediately by correct code, which would then start with the sequence a5 4e 9c 95 68 - which is skipped entirely by objdump.

.. what are these (bad) instructions for?

I suppose, looking at the other instructions that you show, that you are disassembling a wrong part of the executable and this is not supposed to be code at all but data instead.

You should check the PE/PE++ headers to find the sections, and then only attempt disassembly on a section marked as "code" and/or "executable" in its Characteristics field. Even then, it's possible to start at a 'wrong' position (e.g., in the middle of a longer instruction) or inside data (which may also reside inside a .code section).

  • I've already parsed headers and just trying to disassemble only ".text" section as you said and i think you are correct about that it may not be instruction but data. I wonder if there is an another table or something like that for indicating code parts of the .text section.. Since I'm using linear sweep approach to disassemble(like objdump) these datas between instructions keep me busy with them i think.. – Efe Can Nov 24 '16 at 16:34
  • 1
    @EfeCan: no, linear sweep leads to nothing but pain. objdump is "good enough" for a casual peek at code, but for a more serious disassembler you should take a look at IDA Pro (if I'm not mistaken they still have a free demo which does wonders with a PE). The approach I used for my latest disassembler is function driven – perfectly doable, because the executable I'm investigating was written in an OOP language. – usr2564301 Nov 24 '16 at 16:39
  • 1
    yeah i think i should really consider to swap recursive approach. I'm gonna try IDA to get some more ideas.. – Efe Can Nov 24 '16 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.