Analyse a piece of code using FloatingPoint instructions and calculations

Question

i'm reversing this function doing this floating point calculations which eventually leads to a crucial compare. I need to understand what exactly are the calculations that are made and i want to re-write them in python.

Basically i need to rewrite this whole piece of code because the calculations that happen later on are some sort of key generation, but i cant understand what is the calculation done in here.. I was hoping someone could help me figure this out, while i give all the information i already understand.

The piece of code:

sub     esp, 8
movsd   xmm0, ds:MEMORY1 ; mov 128-bit memory    
movsd   [esp+1437Ch+var_1437C], xmm0    
mov     eax, [ebp+ItirationCounter_var_60]
mov     [ebp+CurrentCounter_var_A8], eax    
cvtsi2sd xmm0, [ebp+CurrentCounter_var_A8] ; Convert to floating point into xmm0    
mov     ecx, [ebp+CurrentCounter_var_A8] ; 32    
shr     ecx, 1Fh        ; 0    
addsd   xmm0, ds:MEMORY2[ecx*8] ; adds 0    
sub     esp, 8    
movsd   [esp+14384h+var_14384], xmm0    
call    msvcrt.fmod            ; returns A / B floating point remainer    
add     esp, 10h    
fstp    [ebp+var_F8]    ; Store ST0 in memory    
movsd   xmm0, [ebp+var_F8]    
ucomisd xmm0, ds:MEMORY3 ; always zero    
lahf                    ; load AH Flag register    
test    ah, 44h    
jp      short CrucialFunction

Ok, so the op-codes \ pieces of code i dont understand are: addsd xmm0, ds:MEMORY2[ecx*8], call fmod and the ucomisd xmm0, [MEMORY1] .

1. I can understand that the fmod function is receiving the arguments from the xmm0 opcodes that happen earlier with the sub esp, 8, but the output is not the same as the output i recieve when doing this function on python.. maybe im missing something.

2. i just dont understand what ucomisd is doing.. and how it effects the ZeroFlag and ParityFlag, so this is an issue...

here is my so-far python implementation i made for this little piece of code (it runs in a loop which is missing for the code-snap i gave, so bare with me please)

MEMORY1 = 0x4024000000000000 #qword_7600D600
MEMORY2 = [0x0, 0x41F0000000000000] #qword_7600D620
MEMORY3 = 0x0

def ToDoublePrecision(value):
    # cvtsi2sd xmm0, value
    return hex(struct.unpack('<Q', struct.pack('<d', value))[0])

def KeyGeneration():
    counter = 0
    while 1:
        counter &= 0xFFFFFFFF # Sanity - maintain 32 bit
        xmm0 += MEMORY2[(counter) >> 0x1F] # This is wrong.. ---- addsd   xmm0, ds:MEMORY2[ecx*8] 
        if(1 == (0x80000000 & counter)):
            pass # I dont know what is going on in here..
        else:
            xmm0 = ToDoublePrecision(counter) # This is a little hack.. the value at MEMORY[0] is 0x0.. so i just decided this should happen

        # This calculation is inaccurate for some reason, and differs a lot from the output on register ST0    
        result = math.fmod(MEMORY1, xmm0)
        ....
        ....

        """
        ZeroFlag = 1, well if the last calculatio is set to zero
        ParityFlag = 1, is set if the last calculation amount of set bits ('1' - ones) is odd
        """

I'm just stuck on these opcodes for quite some time.. if anyone can give me a good reference on the internet which explains how they work or what they do.. That could be super helpful.

An example of an fmod call and its arguments: Just before the CALL msvcrt.fmod instruction, ESP looks like that:

EBP-14384 012FBC30   00000000
EBP-14380 012FBC34   40438000
EBP-1437C 012FBC38   00000000
EBP-14378 012FBC3C   40240000

my ItirationCounter is set to 0x27

and the result of the msvcrt.fmod function into ST0 is Float 9.0000000000000000000, Hex 4002 90000000 00000000

Going to 'var_F8' after the opcode fstp [ebp+var_F8], the value is a QUAD_WORD of 0x40220000 00000000 (Ordered)

After all that, LAHF changes AX to the value 0x220, which means AH is 0x02. the jump is taken.

Answer 1

sub esp, 8 movsd xmm0, ds:MEMORY1 ; mov 128-bit memory movsd [esp], xmm0

Fetch MEMORY1. This will be our denominator in fmod().

mov eax, [ebp+ItirationCounter_var_60] mov [ebp+CurrentCounter_var_A8], eax cvtsi2sd xmm0, [ebp+CurrentCounter_var_A8] ; Convert to floating point into xmm0

Fetch Counter as a floating-point number. This will be our numerator in fmod().

mov ecx, [ebp+CurrentCounter_var_A8] ; 32 shr ecx, 1Fh ; 0 addsd xmm0, ds:MEMORY2[ecx*8] ; adds 0

This part is adjusting for negative numbers. I doubt that the counter will ever be so large, so it is not interesting.

sub esp, 8 movsd [esp], xmm0 call msvcrt.fmod ; returns A / B floating point remainer add esp, 10h

So fmod(counter, memory1).

fstp [ebp+var_F8] ; Store ST0 in memory movsd xmm0, [ebp+var_F8] ucomisd xmm0, ds:qword_7600D5E8 ; always zero lahf ; load AH Flag register test ah, 44h jp short CrucialFunction

The JP branch will be taken only in two cases: that AH is 44h or AH is 00h. The first case happens when the values are equal (i.e. the return from fmod() is zero). The second one happens when the return from fmod() has an odd number of bits set, so 1 2 4 7 8 11...

Your Python version has reversed the numerator and denominator parameters.