Analyse a piece of code using FloatingPoint instructions and calculations

i'm reversing this function doing this floating point calculations which eventually leads to a crucial compare. I need to understand what exactly are the calculations that are made and i want to re-write them in python.

Basically i need to rewrite this whole piece of code because the calculations that happen later on are some sort of key generation, but i cant understand what is the calculation done in here.. I was hoping someone could help me figure this out, while i give all the information i already understand.

The piece of code:

``````sub     esp, 8
movsd   xmm0, ds:MEMORY1 ; mov 128-bit memory
movsd   [esp+1437Ch+var_1437C], xmm0
mov     eax, [ebp+ItirationCounter_var_60]
mov     [ebp+CurrentCounter_var_A8], eax
cvtsi2sd xmm0, [ebp+CurrentCounter_var_A8] ; Convert to floating point into xmm0
mov     ecx, [ebp+CurrentCounter_var_A8] ; 32
shr     ecx, 1Fh        ; 0
sub     esp, 8
movsd   [esp+14384h+var_14384], xmm0
call    msvcrt.fmod            ; returns A / B floating point remainer
fstp    [ebp+var_F8]    ; Store ST0 in memory
movsd   xmm0, [ebp+var_F8]
ucomisd xmm0, ds:MEMORY3 ; always zero
lahf                    ; load AH Flag register
test    ah, 44h
jp      short CrucialFunction
``````

Ok, so the op-codes \ pieces of code i dont understand are: `addsd xmm0, ds:MEMORY2[ecx*8]`, `call fmod` and the `ucomisd xmm0, [MEMORY1]` .

1. I can understand that the fmod function is receiving the arguments from the xmm0 opcodes that happen earlier with the `sub esp, 8`, but the output is not the same as the output i recieve when doing this function on python.. maybe im missing something.

2. i just dont understand what `ucomisd` is doing.. and how it effects the `ZeroFlag` and `ParityFlag`, so this is an issue...

here is my so-far python implementation i made for this little piece of code (it runs in a loop which is missing for the code-snap i gave, so bare with me please)

``````MEMORY1 = 0x4024000000000000 #qword_7600D600
MEMORY2 = [0x0, 0x41F0000000000000] #qword_7600D620
MEMORY3 = 0x0

def ToDoublePrecision(value):
# cvtsi2sd xmm0, value
return hex(struct.unpack('<Q', struct.pack('<d', value))[0])

def KeyGeneration():
counter = 0
while 1:
counter &= 0xFFFFFFFF # Sanity - maintain 32 bit
xmm0 += MEMORY2[(counter) >> 0x1F] # This is wrong.. ---- addsd   xmm0, ds:MEMORY2[ecx*8]
if(1 == (0x80000000 & counter)):
pass # I dont know what is going on in here..
else:
xmm0 = ToDoublePrecision(counter) # This is a little hack.. the value at MEMORY[0] is 0x0.. so i just decided this should happen

# This calculation is inaccurate for some reason, and differs a lot from the output on register ST0
result = math.fmod(MEMORY1, xmm0)
....
....

"""
ZeroFlag = 1, well if the last calculatio is set to zero
ParityFlag = 1, is set if the last calculation amount of set bits ('1' - ones) is odd
"""
``````

I'm just stuck on these opcodes for quite some time.. if anyone can give me a good reference on the internet which explains how they work or what they do.. That could be super helpful.

An example of an fmod call and its arguments: Just before the `CALL msvcrt.fmod` instruction, ESP looks like that:

``````EBP-14384 012FBC30   00000000
EBP-14380 012FBC34   40438000
EBP-1437C 012FBC38   00000000
EBP-14378 012FBC3C   40240000
``````

my `ItirationCounter` is set to `0x27`

and the result of the `msvcrt.fmod` function into ST0 is `Float 9.0000000000000000000`, `Hex 4002 90000000 00000000`

Going to 'var_F8' after the opcode `fstp [ebp+var_F8]`, the value is a QUAD_WORD of `0x40220000 00000000` (Ordered)

After all that, `LAHF` changes `AX` to the value `0x220`, which means `AH` is `0x02`. the jump is taken.

```sub esp, 8 movsd xmm0, ds:MEMORY1 ; mov 128-bit memory movsd [esp], xmm0```

Fetch MEMORY1. This will be our denominator in fmod().

```mov eax, [ebp+ItirationCounter_var_60] mov [ebp+CurrentCounter_var_A8], eax cvtsi2sd xmm0, [ebp+CurrentCounter_var_A8] ; Convert to floating point into xmm0```

Fetch Counter as a floating-point number. This will be our numerator in fmod().

```mov ecx, [ebp+CurrentCounter_var_A8] ; 32 shr ecx, 1Fh ; 0 addsd xmm0, ds:MEMORY2[ecx*8] ; adds 0```

This part is adjusting for negative numbers. I doubt that the counter will ever be so large, so it is not interesting.

```sub esp, 8 movsd [esp], xmm0 call msvcrt.fmod ; returns A / B floating point remainer add esp, 10h```

So fmod(counter, memory1).

```fstp [ebp+var_F8] ; Store ST0 in memory movsd xmm0, [ebp+var_F8] ucomisd xmm0, ds:qword_7600D5E8 ; always zero lahf ; load AH Flag register test ah, 44h jp short CrucialFunction```

The JP branch will be taken only in two cases: that AH is 44h or AH is 00h. The first case happens when the values are equal (i.e. the return from fmod() is zero). The second one happens when the return from fmod() has an odd number of bits set, so 1 2 4 7 8 11...

Your Python version has reversed the numerator and denominator parameters.

• Thanks you very much for your response! but the fmod switched parameters is not the case, i tried all kinds of versions with it but it just failed every time.. Ill check our your answer regardless tomorrow, and see if it works as expected.. but still when the code runs in the assembly in the remote program.. the fmod return values are very different than mine
– 0xAK
Nov 9, 2015 at 19:15
• what about the `addsd xmm0, ds:MEMORY2[ecx*8]` opcode. things dont add up. my python version with your suggestions still dont fit the original executable version. Are you sure its `fmod(counter, MEMORY1)` ? I forgot to add the information that `MEMORY1` values (and `MEMORY2`) both are QUAD_WORDS: `MEMORY1 = 0x4024000000000000` `MEMORY2 = [0x0, 0x41F0000000000000]`
– 0xAK
Nov 10, 2015 at 7:31
• python is probably trying to convert your hex values to floating-point values, not realizing that they are floating-point already. Nov 11, 2015 at 17:56