2

I'm using MinHook to hook a static function with the following signature:

float __usercall RadarManager::Add_Blip@<xmm0>(int position@<ecx>, int shape@<edx>, int colorId, float intensity@<xmm3>, bool isBracket, bool isSmall)

I have confirmed that the register/stack usage is correct using IDA. I also have confirmed that I got the right address using Visual Studio to debug my code. It first goes to the trampoline generated by MinHook, executes the first few opcodes and then jumps to the original function skipping the bytes that were overwritten.

I made my hook function completely empty, but the program still crashes on some other arbitrary address.

My hook function is as follows:

float __declspec(naked) radarAddBlip(const Vector3 &position, int shape, int colorId, float intensity, bool isBracket, bool isSmall)
{
    __asm
    {
        call originalRadarManager_Add_Blip;

        ret;
    }
}

Prologue of function that is being hooked:

.text : 1181F4C0                 push    ebp
.text : 1181F4C1                 mov     ebp, esp
.text : 1181F4C3                 and     esp, 0FFFFFFF8h
.text : 1181F4C6                 sub     esp, 34h
.text : 1181F4C9                 movss   xmm2, dword ptr[ecx + 4]
.text : 1181F4CE                 movss   xmm1, dword ptr[ecx + 8]
.text : 1181F4D3                 movss   dword ptr[esp + 34h + var_20], xmm3
.text : 1181F4D9                 movss   xmm3, dword ptr[ecx]
.text : 1181F4DD                 movaps  xmm0, xmm2
.text : 1181F4E0                 mulss   xmm0, dword_11A6D7AC
.text : 1181F4E8                 mulss   xmm2, dword_11A6D7BC
.text : 1181F4F0                 movaps  xmm4, xmm3
.text : 1181F4F3                 mulss   xmm4, dword_11A6D7A8
.text : 1181F4FB                 mulss   xmm3, dword_11A6D7B8
.text : 1181F503                 addss   xmm4, xmm0
.text : 1181F507                 movaps  xmm0, xmm1
.text : 1181F50A                 mulss   xmm0, dword_11A6D7B0
.text : 1181F512                 mulss   xmm1, dword_11A6D7C0
.text : 1181F51A                 addss   xmm3, xmm2
.text : 1181F51E                 addss   xmm4, xmm0
.text : 1181F522                 push    ebx
.text : 1181F523                 push    esi
.text : 1181F524                 mov     esi, edx
.text : 1181F526                 addss   xmm3, xmm1
.text : 1181F52A                 addss   xmm4, dword_11A6D7B4
.text : 1181F532                 movss   xmm1, ds : dword_11A419F0
.text : 1181F53A                 push    edi
.text : 1181F53B                 mov[esp + 40h + var_14], esi
.text : 1181F53F                 addss   xmm3, dword_11A6D7C4
.text : 1181F547                 mulss   xmm4, xmm1
.text : 1181F54B                 mulss   xmm3, xmm1
.text : 1181F54F                 mulss   xmm1, ds : dword_11A3ED98
.text : 1181F557                 movaps  xmm0, xmm3
.text : 1181F55A                 mulss   xmm0, xmm3
.text : 1181F55E                 movaps  xmm2, xmm4
.text : 1181F561                 mulss   xmm2, xmm4
.text : 1181F565                 movss[esp + 40h + var_30], xmm4
.text : 1181F56B                 movss[esp + 40h + var_2C], xmm3
.text : 1181F571                 addss   xmm2, xmm0
.text : 1181F575                 movaps  xmm0, xmm1
.text : 1181F578                 mulss   xmm0, xmm1
.text : 1181F57C                 movss[esp + 40h + var_28], xmm1
.text : 1181F582                 addss   xmm2, xmm0
.text : 1181F586                 cvtps2pd xmm0, xmm2
.text : 1181F589                 movss[esp + 40h + var_24], xmm2
.text : 1181F58F                 call    _libm_sse2_sqrt_precise
.text : 1181F594                 movss   xmm1, ds : dword_11A41A30
.text : 1181F59C                 cvtsd2ss xmm0, xmm0
.text : 1181F5A0                 comiss  xmm0, ds : dword_11A41E10
.text : 1181F5A7                 jb      loc_1181F62E
.text : 1181F5AD                 cmp     esi, 4
.text : 1181F5B0                 jnz     short loc_1181F621
.text : 1181F5B2                 movss   xmm0, [esp + 34h + var_18]
.text : 1181F5B8                 addss   xmm0, ds : dword_11A41828
.text : 1181F5C0                 cvtps2pd xmm0, xmm0
.text : 1181F5C3                 call    _libm_sse2_sqrt_precise

Epilogue of function being hooked:

.text : 1181F7B7                 cmp[ebp + isBracket], 0
.text : 1181F7BB                 jz      short loc_1181F7F8
.text : 1181F7BD                 mov     esi, dword_11A5F400
.text : 1181F7C3                 push    0
.text : 1181F7C5                 push    dword ptr[esi + 0C4h]
.text : 1181F7CB                 mov     ecx, esi
.text : 1181F7CD                 call    sub_11822750
.text : 1181F7D2                 lea     eax, [esp + 34h + var_4]
.text : 1181F7D6                 push    eax
.text : 1181F7D7                 mov     ecx, esi
.text : 1181F7D9                 call    sub_11822330
.text : 1181F7DE                 push    offset dword_11A6D8B0
.text : 1181F7E3                 mov     ecx, esi
.text : 1181F7E5                 call    sub_11822B00
.text : 1181F7EA                 or edi, 0FF00h
.text : 1181F7F0                 push    edi
.text : 1181F7F1                 mov     ecx, esi
.text : 1181F7F3                 call    sub_11822660
.text : 1181F7F8
.text : 1181F7F8 loc_1181F7F8 : ; CODE XREF : RadarManager__Add_Blip + 2FBj
.text : 1181F7F8                 movss   xmm0, ds : dword_11A41A30
.text : 1181F800                 pop     edi
.text : 1181F801                 pop     esi
.text : 1181F802                 pop     ebx
.text : 1181F803                 mov     esp, ebp
.text : 1181F805                 pop     ebp
.text : 1181F806                 retn

Example of original function being called (Left in some extra opcodes):

.text : 11820270                 mov     eax, [ecx + 4]
.text : 11820273                 mov     esi, [eax + edi * 4]
.text : 11820276                 cmp     byte ptr[esi + 24h], 0
.text : 1182027A                 jz      short loc_118202C8
.text : 1182027C                 cmp     dword ptr[esi + 8], 0
.text : 11820280                 jnz     short loc_118202C8
.text : 11820282                 mov     eax, [esi + 4]
.text : 11820285                 movss   xmm3, dword ptr[esi + 34h]; a4
.text : 1182028A                 push    0; a5
.text : 1182028C                 add     eax, 4
.text : 1182028F                 push    0; a5
.text : 11820291                 push    eax; a3
.text : 11820292                 lea     ecx, [esi + 28h]; a1
.text : 11820295                 mov     edx, 4; a2
.text : 1182029A                 call    RadarManager__Add_Blip
.text : 1182029F                 movaps  xmm1, xmm0
.text : 118202A2                 xorps   xmm0, xmm0
.text : 118202A5                 add     esp, 0Ch
.text : 118202A8                 comiss  xmm0, xmm1
.text : 118202AB                 ja      short loc_118202BA
.text : 118202AD                 movss   xmm0, ds : dword_11A41A30
.text : 118202B5                 comiss  xmm1, xmm0
.text : 118202B8                 jbe     short loc_118202BD
.text : 118202BA
.text : 118202BA loc_118202BA : ; CODE XREF : RadarManager__Update + A8Bj
.text : 118202BA                 movaps  xmm1, xmm0
.text : 118202BD
.text : 118202BD loc_118202BD : ; CODE XREF : RadarManager__Update + A98j
.text : 118202BD                 mov     ecx, dword_11A679C4
.text : 118202C3                 movss   dword ptr[esi + 34h], xmm1

Opcodes that are copied to the trampoline (before the jmp)

.text : 1181F4C0                 push    ebp
.text : 1181F4C1                 mov     ebp, esp
.text : 1181F4C3                 and     esp, 0FFFFFFF8h

I've tried changing calling conventions, to making it __declspec(naked) and directly calling it and even manually editing the registry, but to no avail. I'm at the end of my rope and I can't even get it to work when it doesn't even do anything.

To summarize my question: How can I successfully call the original function through my hook function and doing something before calling the original? I'm not required to use MinHook, but so far it has been doing a good job.

  • Have you tried using a debugger on the crash? (Ollydbg, x64dbg etc.) Might reveal some relevent details as to why. Maybe the compiler isn't calling it correctly (possibly because of a prologue on the hook function). I could be on the wrong track though, I'm on my mobile. – Samson Nov 3 '17 at 2:12
  • I will post the information at the crash location asap. I forgot to turn on teamviewer on the PC I'm doing this at, so I can't take a look at it right now. – Neijwiert Nov 3 '17 at 7:50
  • @Samson could it be that it uses __vectorcall? I have never used this calling convention before and only just found out about its existence. But it seems that 'intensity' would then be passed on xmm0? I'm also not sure if I can use this convention. My environment must be using the visual studio 2012 xp build tools. – Neijwiert Nov 3 '17 at 8:38
  • 2
    You have to pass a3, a5 and a6 on the stack, you will have to re-push them before the call to the original function. – Willem Hengeveld Nov 6 '17 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.