Understanding FCOMP instruction and extracted value from address operand

Question

I'm trying to reverse engineer an application made in vb6. At a certain point it compares an input number to a constant number, my goal here is to extract that number, now i found where the comparison is taken place:

FCOMP QWORD PTR DS:[402CB0]

Now I understand that FCOMP Compares the contents of register ST(0) and source value. I don't know if I got this right, but from what I've read DS:[402CB0] is pointer to an address that's holding the source value, but using OllyDbg, and while navigating to that address (Ctrl + g), i found out that the value is DB 00 which is not correct

so my question here is how can I find the real value that is being compared? and is it possible to make the FCOMP compare a constant to a pointer of an integer?

Answer 1

FCOMP Compares the fpu register ST0 with a constant
the constant is a QWORD (Meaning DOUBLE , FLOAT , Etc 8+ bytes wide )

ollydbg can show both the ST0 register and decipher the contents of the CONSTANT .

in your case if shows DB 00 because the constant is probably 0.0 .

and you have not set the dump view mode to appropriate format

the view mode you are looking at is Disassembly (DB is Define Byte 00 is well 0x00 ) .

you may need to change the view mode

first select the dump window then use ctrl+g and then right click select FLOAT .

ollydbg also has a small window between disassembly pane and dump pane which can show both the source and destination contents

in the screen shot below you can observe how 783ef8 is DB 00 in Disassembly window and FLOAT 0.0 in Dump Pane . you can observe how the contents of register pane shows both the src and dest contents you can observe how fpu register window shows the ST0 contents

Answer 2

Understanding the assembly line

FCOMP QWORD PTR DS:[402CB0]

It checks for a qword sized operand. That is 8 bytes while your picture only shows about 3.

This comparison is also a floating point comparison, which is a bit more complex than an integer comparison.

It sets three control registers (C0, C2, C3) depending on the outcome and raises an exception if either values is NaN (could this be some kind of Anti debugging related exception?)

Getting to the actual content at the time of comparison

Although you can access the address now, it is not guaranteed the value you're looking at is the value that will end up being compared to your float.

You should make sure the debugger reaches that address when you inspect it. This can usually be done quite easily by setting a breakpoint (either software - OllyDbgv1's default or hardware - OllyDbgv2's default). Either right click on that assembly line -> Breakpoint -> Toggle or simply hit F2 while it's highlighted.