What to do when IDA does not recognize return value?

Question

I have a function which fetches the Glide screen width and height and passes it to some set-up function (sub_457048).

However, IDA does not recognize that the Glide API function to retrieve the width obviously returns it:

This gets interesting when looking into the assembly code:

I'm not sure how to tell IDA the grSstScreenWidth returns a value into EBX which is then put in EDX: The set-up function looks correct - I never understood that useless mov of EAX into EBX after the height function was called; height stays in EAX, and width is put into EDX:

Am I wrong here? Is IDA wrong? Or is nobody wrong?

Answer 1

You'll need to change the prototype of _grSstScreenWidth() to be a __usercall function with the return value in ebx.

For example:

int __usercall _grSstScreenWidth@<ebx>()

Answer 2

I am probably late for this, but for future visitors: the parameters to sub_457048 are passed in eax and edx. The call to _grSstScreenHeight is returned in eax and then saved in ebx, and finally moved in edx. The call to _grSstScreenWidth is returned in eax.

so sub_457048 will be eax (width) and edx (height) which is correct.

The intermediate step to store the value in ebx, is because it's not known to the caller if _grSstScreenWidth will overwrite edx (as it is allowed to, according to the STDCALL calling convention https://en.wikipedia.org/wiki/X86_calling_conventions#stdcall). On the other hand, _grSstScreenWidth has to preserve ebx.

You're right however, IDA did not seem to see that the value returned by _grSstScreenWidth was to be stored in v4