andps
stands for "Bitwise Logical AND of Packed Single Precision Floating-Point Values", i.e. the contents of the register is interpreted as four separate single precision (32-bit) values and a bitwise AND operation is performed using the second operand as the mask. Since floats do not have bitwise operations defined, the CPU performs it on the raw values in the registers as if they were just a stream of bits. I.e. the result of
andps xmm0, ds:xmmword_108365D0
is similar to the following operations:
xmm0.m128i_u32[0] &= 0x7FFFFFFF;
xmm0.m128i_u32[1] &= 0x7FFFFFFF;
xmm0.m128i_u32[2] &= 0x7FFFFFFF;
xmm0.m128i_u32[3] &= 0x7FFFFFFF;
executed simultaneously (m128i_u32
here represents the contents of the xmm register as an array of four unsigned 32-bit integers).
Since the IEEE floating-point format uses most significant bit (bit 31) as the sign of the value, and this operation clears this bit, obviously this operation makes the number always positive, i.e. it's an equivalent of fabs()
.
The following two operations (subss
and mulss
are of the "Scalar Single-Precision" category, i.e. they use only the low 32 bits of the register as the floating-point value, that's why the decompiler tried to convert the calculation to a single float, however it could not discard the andps
operation because it is performed on the whole register, which resulted in the ugly cast sequence. So, if we replace the &
by fabs
, the "correct" expression is probably this:
v8 = (fabs(v6 - v5) - 4.0) / 6;
float v8 = ( ( float ) ( ( ( uint32_t ) ( v6 - v5 ) ) & 0x7FFFFFFF7FFFFFFF7FFFFFFF7FFFFFFF ) - 4.0 ) * 0.16666667;
This shows that IDA was right. However, the constant used for AND is too large according to my compiler. What is the solution to this?