I'm writing a small utility library for hooking functions at run time. I need to find out the length of the first few instructions because I don't want to assume anything or require the developer to manually input the amount of bytes to relocate and overwrite.
There are many great resources to learn assembly but none of them seem to go into much detail on how assembly mnemonics get turned into raw binary instructions.
If you want to understand the instruction encodings in detail you need to study Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2 (Instruction Set Reference, A-Z). Be aware that Intel IA-32 and AMD64 are very complicated instruction sets and in order to hook a function which is not specifically designed to be hooked by injecting a jump you will run into a great number of different instructions. There is no guarantee that the function even has a stack frame set up.
There are libraries which can do the disassembly and hooking for you, such as Detours by Microsoft Research.
You can use a disassembler to go from binary opcodes to assembly code.
For example ndisam command is able to do this.
If you have the following binary opcodes (hex view of file):
31C0FFC0C3
You will get the following output when disassembling it with ndisasm:
00000000 31C0 xor ax,ax
00000002 FFC0 inc ax
00000004 C3 ret
Where the first column is the file offset, the second is the binary opcodes and the final row is the assembly code.
You could then get the second column and get the string length of it and divide by 2 and you would have the length of the instruction in bytes.
A lot of people have mentioned the Intel manuals, which are an invaluable reference, but quite hefty. I'd suggest looking at this OSDev wiki page to get an idea of how the instructions are encoded on a simpler level.
For all practical instruction-length-finding problems, I would advise using a disassembler.
Function hooking is an interesting challenge. This MSDN blog explains some of the difficulties well. Depending the requirements, it might be preferable to use the operating system's debugging functionality to attach to the process, "break" on functions, and implement your hook in a separate process.
This CodeProject article is an excellent high-level view of x86's instruction format (with diagrams!). After reading this, more detailed references will make more sense.
Due to many years of backwards-compatible evolution, the x86 instruction format is quite complicated, with all sorts of optional prefixes and instruction-dependent fields, so it is a bit tricky to work out the instruction length. If you want something robust, I would advise adapting existing software rather than rolling your own. But understanding these concepts will of course be very helpful.
The ground truth on instruction decoding can be found in the processors manual for software developers. Assembler authors need to know this, so the information is there. For Intel, it's in the beginning of Volume 2A (I think, I lost track since they smushed all the manuals into one PDF). There's a big table that defines how prefixes are encoded, how opcodes are encoded, and how operands are encoded. It's not the easiest reading, but it's there...
The IA-32 Intel® Architecture Software Developer’s Manual Vol. 2 in all its mind-numbing glory.
You can use the capstone library to disassemble the instructions stating at the address you are looking at. It has the ability to determine for you how long the instruction is in bytes, and then you can use that in your code.
There are many great resources to learn assembly but none of them seem to go into much detail on how assembly mnemonics get turned into raw binary instructions.
The mnemonics aren't 'turned in' to raw binary instructions, they are the raw binary instructions. It is a human readable representation of the actual hex bytes that are encoding the instructions of the (first generation language) machine code. We typically talk about these bytes by their mnemonics for clarity.
This is unlike assembly (so, second generation language such as gas or masm) or C (third generation language). Both assembly and higher generation languages are source files with a character encoding (such as UTF-8) that encodes the letters of the source code. In the case of assembly, it needs to be assembled to raw x86 bytes. The raw bytes can be disassembled by IDA into a readable form; it's called disassembly because it's presenting the mnemonic representation encoded in a character set like UTF-8 and displayed to the screen (which resembles an assembly language (a second generation language) that would be assembled to machine code -- in this case it's obviously not going to be assembled and isn't really an assembly language on it's own -- it's showing you what the assembly that was compiled to the machine code would have been).
x86 encoding looks like this (I created a diagram):
By the same token, CIL bytecode is a 1st generation language because the opcodes run directly on the virtual machine -- a virtual CPU. This means that the CIL bytecode does not need to be assembled or compiled from a source file with a source character set. ilasm shows a disassembled, mnemonic form of this bytecode.
