Does V-Table patching (via DLL injection) only work for COM objects/classes?

Question

Okay so not too long ago i was introduced to the concept of hooking/detouring, it took me some time to get my head around, but despite the lack of information available, eventually i was able to hook BeginScene and EndScene using IDA, MS Detours and one of the sample d3d9 applications for testing.

But i realised, that was just one way to do it, so i decided that i'd like to also become familiar with the other methods i'd seen used.

This led me to do some studying on V-Tables, and more specifically, a practice that has been described as dynamic patching of V-Tables.

I found that the typical way of doing this for DirectX is to create a "Dummy device" Then alter it's V-Table, which supposedly affects all instances of the device class.

I went on to try create a scaled down environment using my own class with a single virtual function and some code that instantiated said class, then continuously called said function.

I then injected some code that had access to a copy of this class, it created a new instance and attempted to patch the V-Table function pointer with that of another function.

But i soon found that the way i'd set it up just didn't work as i'd expected. I'm quite certain it modified the V-Table, but only for that instance of the class. (i'd tested this by combining the code into a single program which simply called the function, grabbed the V-Table, modified it to point to another function, then called the function again to confirm it had been swapped)

So i wonder, what is it about DirectX that makes it possible to patch the original V-Table using a new instance of a device class/object?

Is hot patching something specific to COM, or is there some other explanation, maybe i have it all wrong?

(EDIT) Here's what i'm trying to do.

Target:

using namespace std;

class A {
public:
    virtual void doThing() {
        cout << "Class A doThing" << endl;
    }

};


void dynamicOverride() {
    cout << "Function replaced" << endl;
}


int main()
{

    A* a = new A();
    while (1)
    {
        a->doThing();
        Sleep(2000);
    }

    system("pause");
    return 0;
}

Code to inject:

using namespace std;

class A {
public:
    virtual void doThing() {
        cout << "Class A doThing" << endl;
    }

};


void dynamicOverride() {
    cout << "Function replaced" << endl;
}


void MainThread()
{
    A* a = new A();
    void** vtable = *(void***)a;

    DWORD curProtection;
    VirtualProtect(&vtable[0], 4, PAGE_EXECUTE_READWRITE, &curProtection);
    vtable[0] = dynamicOverride;
}

I've seen this done in code that hooks various d3d9 functions, most refer to it as creating a dummy device. But in this case when i try use a class in a DLL and inject the code to modify the V-Table for that class. It never affects the instance of that class in the target process.

Answer 1

You are absolutely on the right track

But,

A* a = new A();
...
a->doThing()

Is not going to bother with the vtable at all. Because a can only be of A type. MSVC will still create the vftable, but in that context a->doThing() will just become A::doThing()

Here, I fixed up your example into my own to demonstrate that you were correct, you just needed to call doThing through some level of inheritance to trigger the vftable lookup.

The A_vftable struct also provides a better representation of what the vtable would be if you could see it.

class Base {
public:
    void virtual VirtualMethod1() = 0;
    void virtual VirtualMethod2() = 0;
};

class A : public Base {
public:
    A() {
        printf("A Constructor\n");
    }

    void virtual VirtualMethod1() {                 
        printf("VirtualMethod1\n");
    }

    void virtual VirtualMethod2() {
        printf("VirtualMethod2\n");  
    }
};

struct A_vftable {
    void(*VirtualMethod1)(A* self);
    void(*VirtualMethod2)(A* self);
};

int main() {

    A a;
    A_vftable* vtable = *(A_vftable**)&a;

    vtable->VirtualMethod1(&a);
    a.VirtualMethod1();

    DWORD oldprotect;
    VirtualProtect(vtable, sizeof(vtable), PAGE_EXECUTE_READWRITE, &oldprotect);
    vtable->VirtualMethod1 = vtable->VirtualMethod2;
    VirtualProtect(vtable, sizeof(vtable), oldprotect, NULL);

    a.VirtualMethod1(); // calls A::VirtualMethod1()
    Base* b = &a;
    b->VirtualMethod1(); // calls our redirected function

    return 0;
}

Oh and void dynamicOverride() actually has different arguments to virtual void doThing().

doThing() is actually void __thiscall doThing(A* this)

since dynamicOverride() is missing the arg here, it's bad, but not catastrophic because in a __thiscall the first arg is passed through ECX.

But not knowing this is a good way to blow the stack, and yourself up later down the line.

Always replace a hooked function with one that handles the args/calling convention exactly.