Randomly picking up a x86 register for an instruction

Question

I came around a piece of malware which i am analyzing and have found that it uses some kind of math to randomly selecting a register for a specific instruction

Which i don't understand how this operation is calculated depend on what?

Here is an example of what i mean

let's say that i wanted to randomly pick up a register for the instruction

ADD DWORD PTR DS:[0],EAX

We know the opcode for this instruction is 01 05 00 00 00 00

The bold number represents the register for this instruction

05 == EAX 0D == ECX

To better explain this here is the instruction with all the registers

0041580B    0105 00000000   ADD DWORD PTR DS:[0],EAX
00415811    010D 00000000   ADD DWORD PTR DS:[0],ECX
00415817    0115 00000000   ADD DWORD PTR DS:[0],EDX
0041581D    011D 00000000   ADD DWORD PTR DS:[0],EBX
00415823    0125 00000000   ADD DWORD PTR DS:[0],ESP
00415829    012D 00000000   ADD DWORD PTR DS:[0],EBP
0041582F    0135 00000000   ADD DWORD PTR DS:[0],ESI
00415835    013D 00000000   ADD DWORD PTR DS:[0],EDI

The malware uses a register index starting from 0 (EAX) till 7 (EDI)

The number is get SHLed first with the number 3 then it is ORed with 5 to get the right register opcode. So my question is how the author came to the conclusion of that?

I would say that SHL REG,3 equals REG*8 that is the number of max registers? but why do we need to OR it with 05? is it because the starting opcode of this instruction is 05?

Does anybody have a better explanation for this? or any hint words for a better comprehend?

Answer 1

To better understand this, you need to study instruction encoding formats i.e. x86 for this question.

An x86 instruction looks like this

+----------------------+--------+--------+-----+--------------+-----------+
| Instruction prefixes | Opcode | ModR/M | SIB | Displacement | Immediate |
+----------------------+--------+--------+-----+--------------+-----------+
|          0-4         |   1-3  |   0-1  | 0-1 |      0-4     |    0-4    |
+----------------------+--------+--------+-----+--------------+-----------+

The numbers on the second row indicates the length in bytes of the corresponding part.

For the instruction,

010D 00000000   ADD DWORD PTR DS:[0],ECX

there is no instruction prefix. The opcode for ADD is 01 (Check here)

The second byte of the instruction i.e ModR/Mis 0D. The ModR/M byte provides addressing information about the instruction. It specifies whether an operand is in a register or in memory; if it is in memory, then fields within the byte specify the addressing mode to be used.

The ModR/M byte can be broken down into

+-----+------------+-----+
| Mod | Reg/Opcode | R/M |
+-----+------------+-----+
|  2  |      3     |  3  |
+-----+------------+-----+

Here the numbers on the second row indicates the length in bits of the corresponding parts.

The Mod field (2 bits) combines with the R/M field (3 bits) to form 32 possible values 8 registers and 24 addressing modes.

The Reg/Opcode field (3 bits) specifies either a register number or three more bits of opcode information; the r/m field (3 bits) can specify a register as the location of an operand, or it can form part of the addressing-mode encoding in combination with the Mod field.

Now, convert the ModR/M i.e 0D to binary. You would get.

+-----+------------+-----+
| Mod | Reg/Opcode | R/M |
+-----+------------+-----+
|  00 |     001    | 101 |
+-----+------------+-----+

The Mod and R/M fields are 00 and 101 respectively. This indicates displacement only addressing mode. See the table below.

For all the instructions this mode of addressing is used, hence the reason for ORing with 5 (in binary 101) to set that particular bit pattern.

Coming to the Reg/Opcode field, this indicates a register.
001 is the register index for ECX.

For the first instruction i.e
0105 00000000 ADD DWORD PTR DS:[0],EAX
this field is 000 standing for EAX. You can check by converting 05 to binary.

See more in the table below taken from here.

So basically the register value was SHLed with 3 to move it to the correct position. The Reg/Opcode field is 3 bits from the right.

Finally the last 4 bytes are 00000000. This represents the displacement which is zero in this example.