Can't find SetCooperativeLevel call in disassembled program

Question

I am trying to make some changes to the setup program that installs the game "Emperor: Battle for Dune". If you want to look at the EXE I am working on, you can download a copy of this setup program from the official Install Fix patch here: http://dune2k.com/Duniverse/Games/Emperor/Downloads/Patches (download the "Install Fix:" entry and it is Setup.exe).

For some reason, the developers decided to make the installer a fullscreen application and so I am trying to modify it to instead run in a window. It uses the DirectDraw API (not entirely sure what version but I think it is 7 or 8). Thus, I know that the function "SetCooperativeLevel" (https://docs.microsoft.com/en-us/windows/desktop/api/ddraw/nf-ddraw-idirectdraw7-setcooperativelevel) is used to initialize the DDraw application in fullscreen or windowed mode.

My problem is that I tried to disassemble the EXE using both IDA Pro 7.0 and OllyDbg 2.01 but neither of them have been able to find any calls to that function. Can anyone tell me if there is anything I can do to help one of these programs find this call? Alternatively, is it possible for a DDraw program to go into fullscreen mode without the SetCooperativeLevel function?

Edit: I attempted to use Defragger's suggestion below. Unfortunately, it didn't work. I modified the code he pointed out as per below to simply change the dimensions to 800x600 but it did not change anything. The window is still in fullscreen and doesn't appear different at all:

.text:004619BF                 mov     esi, ds:GetSystemMetrics
.text:004619C5                 push    258h            ; yBottom
.text:004619CA                 push    320h            ; xRight
.text:004619CF                 push    0               ; yTop
.text:004619D1                 push    0               ; xLeft
.text:004619D3                 push    offset Rect     ; lprc
.text:004619D8                 call    ds:SetRect

I found that I could only modify the "yBottom" and "xRight" parameters safely. If I modified "yTop" or "xLeft", the program crashed.

Answer 1

There's also the possibility to set a screen to full screen with the function SetRect

I found this spot on the mentioned executable:

0x004619bf mov esi, dword [sym.imp.USER32.dll_GetSystemMetrics]
0x004619c5 push 1
0x004619c7 call esi
0x004619c9 push eax
0x004619ca push 0
0x004619cc call esi
0x004619ce push eax
0x004619cf push 0
0x004619d1 push 0
0x004619d3 push 0x48e2f8
0x004619d8 call dword [sym.imp.USER32.dll_SetRect]

The flag 0 and 1 for GetSystemMetrics are called SM_CYSCREEN and SM_CXSCREEN and will make GetSystemMetrics return the max x and y pixels that are finally used for SetRect

EDIT:

I used radare2 to find this spot: /c call~SetRect In ollydbg or any other dynamic tool you can simply add a breakpoint at the top of SetRect and wait until it gets fired to change it's arguments.

EDIT2:

I think i missed some spots with my search /c. Idapro counts four xrefs to SetRect. You could try to set a breakpoint directly at the top of SetRect to see who's calling it. If you would like to change the behavior of a function completely you can use something like https://frida.re i think something like this would work:

Interceptor.attach (Module.findExportByName("user32.dll", "SetRect"), {
onEnter: function (args) {
    args[2]= args[2].shr(1);
    args[3] = args[3].shr(1);
},
onLeave: function (retval) {});

Save this into some js file and spawn your process with this line after installing frida on your system: frida -f SETUP.EXE -l yourscript.js I hope that will work for you.

Answer 2

My problem is that I tried to disassemble the EXE using both IDA Pro 7.0 and OllyDbg 2.01 but neither of them have been able to find any calls to that function.

It won't be called directly because this is a method and not a function. In other word, IDirectDraw instance contains an array of pointers to code (it's called a virtual table), when a call is made, the generated code is an indirect call using the instance. To figure out where this method is called, you have several methods:

• Dynamically: Identify to address to the method and set a breakpoint to it, then you can get the caller when the breakpoint is triggered.
• Statically, find the call to DirectDrawCreateEx and track lplpDD parameter, when the vtable is dereferenced and it's followed by an indirect call with offset (method number * sizeof(void*)), see if the destination is the method you're looking for.
• Lazily, simply use DxWnd :)

I hope it can help you.