1

Okay so not too long ago i was introduced to the concept of hooking/detouring, it took me some time to get my head around, but despite the lack of information available, eventually i was able to hook BeginScene and EndScene using IDA, MS Detours and one of the sample d3d9 applications for testing.

But i realised, that was just one way to do it, so i decided that i'd like to also become familiar with the other methods i'd seen used.

This led me to do some studying on V-Tables, and more specifically, a practice that has been described as dynamic patching of V-Tables.

I found that the typical way of doing this for DirectX is to create a "Dummy device" Then alter it's V-Table, which supposedly affects all instances of the device class.

I went on to try create a scaled down environment using my own class with a single virtual function and some code that instantiated said class, then continuously called said function.

I then injected some code that had access to a copy of this class, it created a new instance and attempted to patch the V-Table function pointer with that of another function.

But i soon found that the way i'd set it up just didn't work as i'd expected. I'm quite certain it modified the V-Table, but only for that instance of the class. (i'd tested this by combining the code into a single program which simply called the function, grabbed the V-Table, modified it to point to another function, then called the function again to confirm it had been swapped)

So i wonder, what is it about DirectX that makes it possible to patch the original V-Table using a new instance of a device class/object?

Is hot patching something specific to COM, or is there some other explanation, maybe i have it all wrong?

(EDIT) Here's what i'm trying to do.

Target:

using namespace std;

class A {
public:
    virtual void doThing() {
        cout << "Class A doThing" << endl;
    }

};


void dynamicOverride() {
    cout << "Function replaced" << endl;
}


int main()
{

    A* a = new A();
    while (1)
    {
        a->doThing();
        Sleep(2000);
    }

    system("pause");
    return 0;
}

Code to inject:

using namespace std;

class A {
public:
    virtual void doThing() {
        cout << "Class A doThing" << endl;
    }

};


void dynamicOverride() {
    cout << "Function replaced" << endl;
}


void MainThread()
{
    A* a = new A();
    void** vtable = *(void***)a;

    DWORD curProtection;
    VirtualProtect(&vtable[0], 4, PAGE_EXECUTE_READWRITE, &curProtection);
    vtable[0] = dynamicOverride;
}

I've seen this done in code that hooks various d3d9 functions, most refer to it as creating a dummy device. But in this case when i try use a class in a DLL and inject the code to modify the V-Table for that class. It never affects the instance of that class in the target process.

  • There's nothing special about directx. Could you add a minimal code example? – NirIzr Aug 24 '18 at 16:14
  • Edited, hopefully i explained clearly enough. – Michael Strobel Aug 24 '18 at 17:07
  • For completeness, please include expected result and actual result. The issue (as far as I understand) is with multiple objects, which isn't reflected in the code example. To address your actual issue: Have you tried disassembling your example code and find out what's wrong? – NirIzr Aug 24 '18 at 18:10
  • In your example, if the code us compiled with optimization enabled, it us quite likely that the compiler knows that a points to an A object, and devirtualizes the call. It means there will be a straight call to A::doThing instead of a reference through the vtable. Another possibility is that the compiler caches the result of the VTable lookup of the first invocation of doThing and reuses the cached address. – Michael Karcher Aug 24 '18 at 21:55
  • I edited again with a exact example of what i was trying to do. I expected both instances of A class to be treated by the target process as actually being the same class, from what i know all class instances share a V-Table, and i'd like to be able to re-write the V-Table function pointer from the DLL. As i said, i have seen code that does this but i'd like to be able to re-create it in my own scenario and understand how/why it works. – Michael Strobel Aug 25 '18 at 2:23
3

You are absolutely on the right track

But,

A* a = new A();
...
a->doThing()

Is not going to bother with the vtable at all. Because a can only be of A type. MSVC will still create the vftable, but in that context a->doThing() will just become A::doThing()

Here, I fixed up your example into my own to demonstrate that you were correct, you just needed to call doThing through some level of inheritance to trigger the vftable lookup.

The A_vftable struct also provides a better representation of what the vtable would be if you could see it.

class Base {
public:
    void virtual VirtualMethod1() = 0;
    void virtual VirtualMethod2() = 0;
};

class A : public Base {
public:
    A() {
        printf("A Constructor\n");
    }

    void virtual VirtualMethod1() {                 
        printf("VirtualMethod1\n");
    }

    void virtual VirtualMethod2() {
        printf("VirtualMethod2\n");  
    }
};

struct A_vftable {
    void(*VirtualMethod1)(A* self);
    void(*VirtualMethod2)(A* self);
};

int main() {

    A a;
    A_vftable* vtable = *(A_vftable**)&a;

    vtable->VirtualMethod1(&a);
    a.VirtualMethod1();

    DWORD oldprotect;
    VirtualProtect(vtable, sizeof(vtable), PAGE_EXECUTE_READWRITE, &oldprotect);
    vtable->VirtualMethod1 = vtable->VirtualMethod2;
    VirtualProtect(vtable, sizeof(vtable), oldprotect, NULL);

    a.VirtualMethod1(); // calls A::VirtualMethod1()
    Base* b = &a;
    b->VirtualMethod1(); // calls our redirected function

    return 0;
}

Oh and void dynamicOverride() actually has different arguments to virtual void doThing().

doThing() is actually void __thiscall doThing(A* this)

since dynamicOverride() is missing the arg here, it's bad, but not catastrophic because in a __thiscall the first arg is passed through ECX.

But not knowing this is a good way to blow the stack, and yourself up later down the line.

Always replace a hooked function with one that handles the args/calling convention exactly.

| improve this answer | |
  • I got this to work using something similar, Base class with pure virtual functions, and a class that inherited from said class. But i did it in a DLL and kinda assumed it was a problem with static linking or something. I will try put this into a static library and compile, if it works as you say with a level of inheritance, then i'll accept this as an answer as it's quite informative. – Michael Strobel Aug 27 '18 at 14:12
  • I tried to implement this, it works when all logic is contained in the same program, but when i put the interface and concrete class into a static lib, then try use it in both the target program, and in the dll i'm using to patch the V-Table. It seems i still have the same problem. So far the only way I've got it to work is by loading the library dynamically. – Michael Strobel Aug 28 '18 at 7:15
  • Of course you're going to have to load whatever class you are trying to modify from the same location as the thing you're trying to hook. This was pointed out to you in previous comments. You can't just make an identical class A and expect a vtable hook to work on someone else's class A. Is that what you mean by static? Just post a larger example of what doesn't work. – Abigail Aug 28 '18 at 9:27
  • Sorry, i thought it was somewhat implied with the code i gave. The target binary and the DLL to patch the V-Table would have to be compiled at different times, just using identical copies of the class, so from what i gather it is only possible with a shared library, and the module has to be loaded into memory space of the target process? – Michael Strobel Aug 28 '18 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.