How return type works [closed]

Question

I declared this

int returnDummy() {
    return 0x100;
}

and compile as a static libray. What result will be returned if I declare the prototype like this,

bool returnDummy();

and link upper static library and call that?

Does it simply casts 0x100 to true, or does it read just lowest 1 byte 0x00 and return false?

Answer 1

That's undefined behavior. Aka the compiler/linker is allowed to break all the rules or make up their own.

However the typical function call ABI will put the return value into EAX if it fits (which both bool and int will). With bool either being defined as 0 for false and non zero for true or as zero for false and a specific value (like 0xffffffff or 1) for true.

So after the function call the return register will contain 0x100 but the compiler could assume it was a specific value (if the ABI said so) and then use that assumption for further optimization.

For example if true was defined as 0xffffffff then a ternary returnDummy()?val1:val2 can be implemented as

int mask = returnDummy();
res = (mask&val1) | (~mask&val2);