Replacing a function in a 1995 Windows 32-bit binary using Frida causes crash

Question

I am working with a 1995 Windows 32-bit binary compiled with Watcom, which uses the Watcom calling convention. I am trying to replace a simple string length calculation function in the binary using Frida. The decompiled version of the function is:

uint calculate_string_length(char *input_string)

{
  char *current_ptr;
  char current_char;
  char *next_ptr;
  
  current_ptr = input_string;
  do {
    next_ptr = current_ptr + 1;
    current_char = *current_ptr;
    current_ptr = next_ptr;
  } while (current_char != '\0');
  return (uint)(next_ptr + (-1 - (int)input_string));
}

And the assembly code is:

        00452420 53              PUSH       EBX
        00452421 56              PUSH       ESI
        00452422 8b f0           MOV        ESI,input_string_EAX
        00452424 8b de           MOV        EBX,ESI
                             LAB_00452426                                    XREF[1]:     00452429(j)  
        00452426 ac              LODSB      current_ptr
        00452427 0a c0           OR         input_string_EAX,input_string_EAX
        00452429 75 fb           JNZ        LAB_00452426
        0045242b 8b c6           MOV        input_string_EAX,current_ptr
        0045242d 2b c3           SUB        input_string_EAX,EBX
        0045242f 48              DEC        input_string_EAX
        00452430 5e              POP        current_ptr
        00452431 5b              POP        EBX
        00452432 c3              RET

I used Frida's Interceptor.attach to log the function inputs and outputs successfully. I then wrote my own JavaScript implementation using Frida's NativeCallback:

var newImplementation = new NativeCallback(function () {
    console.log("######################################")
    const input_string = ptr(this.context.eax);
    const str1 = Memory.readCString(input_string);
    console.log("calculate_string_length In:", str1)
    const length = str1.length;
    console.log("calculated length: ", length)
    return length
}, 'uint32', []);

Interceptor.replace(ptr("0x00452420"), newImplementation);

However, after processing a specific input string, the program crashes. I have checked the address where it crashes, and it's in another function that calls the replaced function. I have already replaced around 10 functions in this binary without any issues, and this is the first function causing problems.

I also tried to change the calling convention to stdcall as it's more similar to Watcom's calling convention and that helped a bit. The program was able to run further but it eventually crashes again.

I also asked the question at: https://github.com/frida/frida/issues/2502

Answer 1

Your hook probably clobbers some register which is used by the caller. Watcom fastcall is a bit special and different from stdcall. Maybe try saving and restoring all registers which are not used for the return value (not sure if Frida can do this).