Returning value from faux DLL

Question

A bit of background; I'm trying to inject my own code into the old game SimTower with the ultimate goal of reverse engineering it in the same way that OpenRCT2 was created.

Since SimTower is distributed as a 16-bit NE executable I'm having a bit of trouble with injecting my own DLL and changing the main method to my own (as in this article https://bwrsandman.wordpress.com/2014/12/27/first-steps-to-reverse-engineering-an-executable/), but I think I've found another way.

One of the first things that SimTower does is to run WAVEMIXINIT from WAVMIX16.DLL. As far as I can tell, if the return value is 0 it shows a dialog box and disables the sound.

My plan now is to simply create a faux wavmix16 dll that will (for now) always return 0 from that function. That should allow me to get an entry point to run my own code, and get the game started.

typedef void *HANDLE;

HANDLE __far __pascal WAVEMIXINIT() {
  // while (1) {;};
  return 0;
}

void __far __pascal STUB4() {}
void __far __pascal STUB5() {}
void __far __pascal STUB6() {}
void __far __pascal STUB7() {}
void __far __pascal STUB9() {}
void __far __pascal STUB10() {}
void __far __pascal STUB11() {}
void __far __pascal STUB12() {}

Compiling this into a dll and naming it wavmix16.dll seems to work, and I can see that the call is coming in to my function, since if I uncomment the while loop the program will be stuck inside it.

For some reason though, the program crashes as soon as I get to my return statement.

(wine)

Unhandled page fault on read access to 0x0000fa30 at address 0x148f:0x0000000c (thread 002b)

(Windows XP)

This is the assembly of my DDL function that gets called. I'm a little bit unsure of why there is more stuff than just a mov and retf (like the nop?), but I'm new to all this so maybe that's normal.

This is the assembly of the original DLL function:

Really appreciate any help, thank you!

Answer 1

Your faux function is not fixing up the stack on the epilog of the function. It should have

pop ds
pop bp
dec bp

at the end by the calling convention.

If you look at the original function it fixes up the stack correctly.

Let's say the stack pointer had the value of 1000h coming into the function (after CS:IP is already pushed on the stack for the far call). You would then have in the original function

; sp = 1000h
mox ax, ds     
...
push bp        ; sp = 0ffeh
mov bp, sp     ; bp = 0ffeh
...
lea sp, [bp-2] ; sp = 0ffch
pop ds         ; sp = 0ffeh
pop bp         ; sp = 1000h
retf

---

To make the compiler used in the question, the Digital Mars compiler dmc.exe, generate these instructions when returning from a far function, pass the flag -Wm which will "Generate INC BP / DEC BP to mark far stack frames".