6

A bit of background; I'm trying to inject my own code into the old game SimTower with the ultimate goal of reverse engineering it in the same way that OpenRCT2 was created.

Since SimTower is distributed as a 16-bit NE executable I'm having a bit of trouble with injecting my own DLL and changing the main method to my own (as in this article https://bwrsandman.wordpress.com/2014/12/27/first-steps-to-reverse-engineering-an-executable/), but I think I've found another way.

One of the first things that SimTower does is to run WAVEMIXINIT from WAVMIX16.DLL. As far as I can tell, if the return value is 0 it shows a dialog box and disables the sound.

My plan now is to simply create a faux wavmix16 dll that will (for now) always return 0 from that function. That should allow me to get an entry point to run my own code, and get the game started.

typedef void *HANDLE;

HANDLE __far __pascal WAVEMIXINIT() {
  // while (1) {;};
  return 0;
}

void __far __pascal STUB4() {}
void __far __pascal STUB5() {}
void __far __pascal STUB6() {}
void __far __pascal STUB7() {}
void __far __pascal STUB9() {}
void __far __pascal STUB10() {}
void __far __pascal STUB11() {}
void __far __pascal STUB12() {}

Compiling this into a dll and naming it wavmix16.dll seems to work, and I can see that the call is coming in to my function, since if I uncomment the while loop the program will be stuck inside it.

For some reason though, the program crashes as soon as I get to my return statement.

(wine)

Unhandled page fault on read access to 0x0000fa30 at address 0x148f:0x0000000c (thread 002b)

(Windows XP)

This is the assembly of my DDL function that gets called. I'm a little bit unsure of why there is more stuff than just a mov and retf (like the nop?), but I'm new to all this so maybe that's normal.

This is the assembly of the original DLL function:

Really appreciate any help, thank you!

10
  • Did you check if you are using the correct parameters and calling convention? This seems like a clean-up error.
    – Nordwald
    Nov 7, 2017 at 5:52
  • As far as I can tell I am. Looking at the header file for the wavemix library I can see that the function is defined as HANDLE WINAPI WaveMixInit(void);. HANDLE is defined as a void pointer which I define in the same way in my code. WINAPI, if I understand this article correct is the same as my __far __pascal... I'll add the decompiled function from the original DLL and see if I can spot any differences... Nov 7, 2017 at 7:26
  • To me it seems like my function is pushing 2 values to the stack before returning, and that the original DLL is pushing 6 and popping 4 values (leaving 2 again) before returning, which to me indicates that the calling conventions should be the same... Nov 7, 2017 at 7:41
  • WINAPI should resolve to __stdcall (on modern systems). On another node, your function does not restore the orginal bp and sp. Really strange convention
    – Nordwald
    Nov 7, 2017 at 8:21
  • So, I'm compiling a 16-bit DLL with an ancient compiler since I haven't found any modern system that outputs 16-bit code :) I'm following this guide from DigitalMars and are using their compiler Nov 7, 2017 at 8:36

1 Answer 1

6

Your faux function is not fixing up the stack on the epilog of the function. It should have

pop ds
pop bp
dec bp

at the end by the calling convention.

If you look at the original function it fixes up the stack correctly.

Let's say the stack pointer had the value of 1000h coming into the function (after CS:IP is already pushed on the stack for the far call). You would then have in the original function

; sp = 1000h
mox ax, ds     
...
push bp        ; sp = 0ffeh
mov bp, sp     ; bp = 0ffeh
...
lea sp, [bp-2] ; sp = 0ffch
pop ds         ; sp = 0ffeh
pop bp         ; sp = 1000h
retf

To make the compiler used in the question, the Digital Mars compiler dmc.exe, generate these instructions when returning from a far function, pass the flag -Wm which will "Generate INC BP / DEC BP to mark far stack frames".

4
  • 1
    about inc bp/dec bp: blogs.msdn.microsoft.com/oldnewthing/20110316-00/?p=11203
    – Igor Skochinsky
    Nov 8, 2017 at 12:44
  • Thank you! I added xor ax,ax, pop ds, pop bp, dec bp, retf as inline assembly in the function and it solved it! Nov 8, 2017 at 16:55
  • That being said, if someone could point me in the right direction to get the DigitalMars compiler dmc.exe to output the correct instructions I would be super happy :) Nov 8, 2017 at 17:03
  • I found it! The flag -Wm will make dmc "Generate INC BP / DEC BP to mark far stack frames". Adding this to the answer for completeness. It could potentially be good to add how to do it in other compilers as well :) Nov 8, 2017 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.