I'm writing a small utility library for hooking functions at run time. I need to find out the length of the first few instructions because I don't want to assume anything or require the developer to manually input the amount of bytes to relocate and overwrite.

There are many great resources to learn assembly but none of them seem to go into much detail on how assembly mnemonics get turned into raw binary instructions.

up vote 18 down vote accepted

If you want to understand the instruction encodings in detail you need to study Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2 (Instruction Set Reference, A-Z). Be aware that Intel IA-32 and AMD64 are very complicated instruction sets and in order to hook a function which is not specifically designed to be hooked by injecting a jump you will run into a great number of different instructions. There is no guarantee that the function even has a stack frame set up.

There are libraries which can do the disassembly and hooking for you, such as Detours by Microsoft Research.

  • 1
    Thanks! That seems to be exactly what I'm looking for. Detours is pretty much the same what I'm doing but I'm reinventing the wheel for educational purposes. – Henry Heikkinen Mar 19 '13 at 23:47
  • Careful, the Detours EULA isn't exactly what one would call liberal. I know it's being referenced everywhere, but the license attached to it disqualifies it for a great many things. – 0xC0000022L Apr 1 '13 at 1:36

You can use a disassembler to go from binary opcodes to assembly code.

For example ndisam command is able to do this.

If you have the following binary opcodes (hex view of file):

31C0FFC0C3

You will get the following output when disassembling it with ndisasm:

00000000  31C0              xor ax,ax
00000002  FFC0              inc ax
00000004  C3                ret

Where the first column is the file offset, the second is the binary opcodes and the final row is the assembly code.

You could then get the second column and get the string length of it and divide by 2 and you would have the length of the instruction in bytes.

  • While this explains how to do it manually on the command line I don't think it is a suitable approach for a utility library that hooks arbitrary code at runtime. If ndisasm is available as a library it might work though, but then you should add that to your answer. – jix Mar 19 '13 at 20:02
  • I've used both the udis86 and opdis libraries for basically this. Both work reasonably well and will (iirc) expose the length of each instruction as a struct member instead of having to parse asm output. – Sean B. Mar 19 '13 at 20:17
  • @SeanB. taking a quick look at udis86, I can see that it has move to github. – Tyilo Mar 19 '13 at 20:22

A lot of people have mentioned the Intel manuals, which are an invaluable reference, but quite hefty. I'd suggest looking at this OSDev wiki page to get an idea of how the instructions are encoded on a simpler level.

For all practical instruction-length-finding problems, I would advise using a disassembler.

Function hooking is an interesting challenge. This MSDN blog explains some of the difficulties well. Depending the requirements, it might be preferable to use the operating system's debugging functionality to attach to the process, "break" on functions, and implement your hook in a separate process.

  • That looks like a great overview. Thanks! – Henry Heikkinen Mar 20 '13 at 12:22

This CodeProject article is an excellent high-level view of x86's instruction format (with diagrams!). After reading this, more detailed references will make more sense.

Due to many years of backwards-compatible evolution, the x86 instruction format is quite complicated, with all sorts of optional prefixes and instruction-dependent fields, so it is a bit tricky to work out the instruction length. If you want something robust, I would advise adapting existing software rather than rolling your own. But understanding these concepts will of course be very helpful.

The ground truth on instruction decoding can be found in the processors manual for software developers. Assembler authors need to know this, so the information is there. For Intel, it's in the beginning of Volume 2A (I think, I lost track since they smushed all the manuals into one PDF). There's a big table that defines how prefixes are encoded, how opcodes are encoded, and how operands are encoded. It's not the easiest reading, but it's there...

  • Hmm, when was that? Even just a few weeks ago they were all nicely separated into separate PDF files ... – 0xC0000022L Apr 1 '13 at 1:35

You can use the capstone library to disassemble the instructions stating at the address you are looking at. It has the ability to determine for you how long the instruction is in bytes, and then you can use that in your code.

http://www.capstone-engine.org/

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.