I'm writing a small utility library for hooking functions at run time. I need to find out the length of the first few instructions because I don't want to assume anything or require the developer to manually input the amount of bytes to relocate and overwrite.
There are many great resources to learn assembly but none of them seem to go into much detail on how assembly mnemonics get turned into raw binary instructions.
If you want to understand the instruction encodings in detail you need to study Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2 (Instruction Set Reference, A-Z). Be aware that Intel IA-32 and AMD64 are very complicated instruction sets and in order to hook a function which is not specifically designed to be hooked by injecting a jump you will run into a great number of different instructions. There is no guarantee that the function even has a stack frame set up.
There are libraries which can do the disassembly and hooking for you, such as Detours by Microsoft Research.
You can use a disassembler to go from binary opcodes to assembly code.
For example ndisam command is able to do this.
If you have the following binary opcodes (hex view of file):
31C0FFC0C3
You will get the following output when disassembling it with ndisasm:
00000000 31C0 xor ax,ax
00000002 FFC0 inc ax
00000004 C3 ret
Where the first column is the file offset, the second is the binary opcodes and the final row is the assembly code.
You could then get the second column and get the string length of it and divide by 2 and you would have the length of the instruction in bytes.
A lot of people have mentioned the Intel manuals, which are an invaluable reference, but quite hefty. I'd suggest looking at this OSDev wiki page to get an idea of how the instructions are encoded on a simpler level.
For all practical instruction-length-finding problems, I would advise using a disassembler.
Function hooking is an interesting challenge. This MSDN blog explains some of the difficulties well. Depending the requirements, it might be preferable to use the operating system's debugging functionality to attach to the process, "break" on functions, and implement your hook in a separate process.
This CodeProject article is an excellent high-level view of x86's instruction format (with diagrams!). After reading this, more detailed references will make more sense.
Due to many years of backwards-compatible evolution, the x86 instruction format is quite complicated, with all sorts of optional prefixes and instruction-dependent fields, so it is a bit tricky to work out the instruction length. If you want something robust, I would advise adapting existing software rather than rolling your own. But understanding these concepts will of course be very helpful.
The ground truth on instruction decoding can be found in the processors manual for software developers. Assembler authors need to know this, so the information is there. For Intel, it's in the beginning of Volume 2A (I think, I lost track since they smushed all the manuals into one PDF). There's a big table that defines how prefixes are encoded, how opcodes are encoded, and how operands are encoded. It's not the easiest reading, but it's there...
You can use the capstone library to disassemble the instructions stating at the address you are looking at. It has the ability to determine for you how long the instruction is in bytes, and then you can use that in your code.
