Detours not cleaning the stack correctly

Question

I'm trying to hook a function using Detours for the first time. I'm new to reversing software and to hooking, so I may have missed something big here.

I'm trying to use this function as a hook:

typedef  int (__thiscall* func_type)(LPVOID*, LPVOID*, DWORD, BOOL);
LPVOID hookaddr = 0;
 int __fastcall testhook(LPVOID* pThis, void* _EDX, LPVOID* object, DWORD hp, BOOL self)
{
    std::cout << "Hooked"  << std::endl;

    func_type originalFunc = (func_type)hookaddr;

    return originalFunc(pThis, object, hp, self);
}

The problem lies in the fact that the hooked function seems to follow __thiscall convention and Detours doesn't seem to be allowing this? I tried mixing __thiscall and __stdcall/__fastcall conventions in my injected dll, but couldn't get anything working. I either end up with a wrong ecx value or a crash due to an invalid esp.

Any idea what I could try here?

Answer 1

I have found the solution to this problem, which is quite simple. I got confused with the hooked function signature and the __fastcall trick to get a non member function to work with the __thiscall convention. In my initial post, the two first arguments in the function signatures are the one passed using ecx and edx, but I forgot about one of the pointer passed using the stack. The correct function definition is the following:

 int __fastcall testhook(LPVOID* pThis, void* _EDX, LPVOID* object, LPVOID* object2, DWORD hp, BOOL self)
{
    std::cout << "Hooked"  << std::endl;

    func_type originalFunc = (func_type)hookaddr;

    return originalFunc(pThis, object, object2, hp, self);
}

As the callee is responsible for cleaning the stack, providing an incorrect number of parameters lead to a stack corruption (in my case, one of the parameter was still on the stack after my function returns).