14

I am using IDA Pro to disassemble a C++ behemoth with 1600+ classes, many of them having pure virtual methods.

Some classes also are made up of multiple base classes, in hierarchies 5+ levels deep.

Ida PRO supports making structures of pointers, to handle vtables, but some of the final classes can have multiple different vtables in the same "slot", due to heavy polymorphism, so how you organize the vtables? How you tell IDA that in this method, or that method, what vtable is actually being refered to?

  • Funny enough I solved this once for a class heavy executable by writing my own disassembler. At its current state it can gather class members and functions from several DLLs. But not a general solution: the code relies extremely on the particular compiler the code was written with. – usr2564301 Jan 20 '16 at 18:55
  • 1
    Why I was downvoted? – speeder Feb 1 '16 at 19:02
  • Athough this question is quite old try using HexRaysPyTools for hexrays – sudhackar Aug 11 '18 at 7:54
22

This depends on the compiler originally used as each creates slightly different layouts. You can find tutorials for most compilers on the net, I'm going to focus MSVC as that's the only one I have experience with, and since it provides a hidden compiler switch printing how classes will be layouted in memory which I'm going to use for illustration.

As you might have already guessed, you have to recreate C++ classes with C structs. This is possible but bothersome, especially if you have as many classes as you said. Some tips to make it less annoying:

  • Focus the ones you're interested in.
  • Start with the "base-most" classes, e.g. the ones at the top of the hierarchy tree. If you mess up something there, fixing it can be expensive later on: Imagine adding a forgotten member to something like a smart pointer base class which is inherited by hundreds of classes, and now you have to remove it from all of those! There's no layout strategy I know of which makes belated fixing easy.
  • You may want to define the structs with C like code in the Local Types subview first and not in the Structures subview. The following samples can be pasted as local types.
  • Simulate the hierarchy you discovered in IDA in a scratch program you quickly coded, compile and load it into IDA to get a nice playground to test your structs with (Make sure to actually instantiate the classes or MSVC typically doesn't include them in the output, even with no optimizations on).
  • If you're unsure about reversing a struct, a constructor typically spoils the vftable and the layout of some members; a function calling it may yield the total size of the object by reserving enough bytes for it.
  • IDA 7.0 helps you a lot by automatically recognizing and marking RTTI data which spoils the class hierarchy and vftables, but it lacks marking which vftable belongs to which base class, an issue the original IDA6 script it seems to be based on didn't have, and that one no longer works in IDA7, shame.
  • IDA 7.2 has even better support for automatically detecting and recreating C++ structures, but the following answer was written with 7.0 in mind and using 7.2 naming.

Depending on how fancy your polymorphism is, your C structs have to be more or less fancy aswell. So let's start with the simple cases first and work our way to the more complicated ones.


No inheritance (base class)

The simplest case does not use any inheritance at all, only some members and (pure) virtual methods to start with:

class Animal {
    int _age;
    Animal() { _age = 0; }
    int getAge() { return _age; }
    virtual void setAge(int value) { _age = value; }
    virtual void makeSound() = 0;
};

MSVC layout:

class Animal    size(8):
        +---
 0      | {vfptr} // pointer to the vftable (s. below)
 4      | _age    // members of Animal
        +---

Animal::$vftable@:
        | &Animal_meta    // ignore meta for our examples
        |  0              // the (pure) virtual method follow
 0      | &Animal::setAge    
 1      | &Animal::makeSound

IDA representation:

struct Animal;
struct Animal_vtbl {
  void (__thiscall *setAge)(Animal *this, int value);
  void (__thiscall *makeSound)(Animal *this);
};
struct Animal_mbrs {
  int _age;
};
struct Animal {
  Animal_vtbl *__vftable;
  Animal_mbrs __members;
};
  • Forward declare the class struct to use it in the this parameter of the vftable.
  • The __thiscall calling convention is required to create class methods in MSVC. It implicitly passes a pointer to the class instance in the ecx register besides all other parameters.
  • It is not required to provide names for the parameters.
  • makeSound will be a purecall, and setAge will be a typical unknown sub modifying our member.
  • Place members in a separate structure for inheritance (s. below).

Single inheritance

Let's quickly breed a Dog inheriting from Animal, implement the makeSound method, and add a new virtual method to set the fur color:

class Dog : public Animal {
    int _furColor;
    virtual void setAge(int value) { _age = value; }
    virtual void makeSound() { cout << "Woof Woof"; }
    virtual void setFurColor(int color) { _furColor = color; }
};

MSVC layout: The Animal base class is simply embedded inside the Dog class. The embedded Animal vftable also grabs all the virtual Dog methods and adds them at its end. The members of Dog appear behind Animals members:

class Dog       size(12):
        +---
 0      | +--- (base class Animal)
 0      | | {vfptr}
 4      | | _age
        | +---
 8      | _furColor
        +---

Dog::$vftable@:
        | &Dog_meta
        |  0
 0      | &Dog::setAge
 1      | &Dog::makeSound
 2      | &Dog::setFurColor // Added behind the Animal methods!

IDA representation: Leaving Animal structures untouched, we add the following:

struct Dog;
struct Dog_vtbl : Animal_vtbl {
  void (__thiscall *setFurColor)(Dog *this, int color);
};
struct Dog_mbrs : Animal_mbrs {
  int _furColor;
};
struct Dog {
  Dog_vtbl *__vftable;
  Dog_mbrs __members;
};
  • Reuse the Animal vftable by letting the Dog vftable inherit from it (inheriting a struct in IDA simply means prepending it), then add the Dog specific virtual functions.
  • The same thing happens with the members, which is why I separated them earlier.

Multiple inheritance

This requires a bit of mind-mangling. For this, we make it possible to kill our dog (sorry if that's cruel to you, I'm bad at creating happy examples):

class Killable {
    bool _isDead;
    virtual void kill() { makeDeathSound(); _isDead = true; }
    virtual void makeDeathSound() = 0;
};

class Dog : public Animal, public Killable {
    int _furColor;
    virtual void setAge(int value) { _age = value; }
    virtual void makeSound() { cout << "Woof Woof"; }
    virtual void setFurColor(int color) { _furColor = color; }
    virtual void makeDeathSound() { cout << "I'll call WWF, bark-blerg"; }
};

MSVC layout: As with the Animal base class, it embeds the second Killable base class into the Dog class aswell, and keeps Dogs members separate. While Dog-specific virtual methods are still merged with the Animal vftable (aka the first base class), Killable related ones are in a separate Killable-related vftable, so we now have:

class Dog       size(20):
        +---
 0      | +--- (base class Animal)
 0      | | {vfptr}
 4      | | _age
        | +---
 8      | +--- (base class Killable)
 8      | | {vfptr}
12      | | _isDead
        | | <alignment member> (size=3) // since _isDead is a 1-byte bool
        | +---
16      | _furColor
        +---

Dog::$vftable@Animal@:
        | &Dog_meta
        |  0
 0      | &Dog::setAge
 1      | &Dog::makeSound
 2      | &Dog::setFurColor // Dog methods still merged with Animal!

Dog::$vftable@Killable@: // All the Killable-related methods in here
        | -8 // offset for `this` pointer in Killable methods to get a Dog pointer
 0      | &Killable::kill
 1      | &Dog::makeDeathSound

IDA representation: We keep a Dog-specific vftable at the start which reuses the Animal vftable internally, as the Dog virtual methods are still appended to Animals vftable. Then Animals members follow as usual. Now, the untouched Killable structs follow, since nothing gets merged into them. At the end, our Dog members follow. If you compare this to the offsets MSVC printed, it makes sense:

struct Killable;
struct Killable_vtbl {
    void (__thiscall *kill)(Killable *this);
    void (__thiscall *makeDeathSound)(Killable *this);
};
struct Killable_mbrs {
    bool _isDead;
};
struct Killable {
  Killable_vtbl* __vftable;
  Killable_mbrs __members;
};

struct Dog;
struct Dog_vtbl : Animal_vtbl {
  void (__thiscall *setFurColor)(Dog *this, int color);
};
struct Dog_mbrs { // No more base Animal members as they're split up now!
  int _furColor; 
};
struct Dog {
  Dog_vtbl *__vftable; // Still contains animal methods.
  Animal_mbrs __members_Animal; // Animal members come here separately.
  Killable_vtbl *__vftable_Killable;
  Killable_mbrs __members_Killable;
  Dog_mbrs __members;
};

IDA 7.2 uses slightly different naming for vftables participating in multiple inheritance. I find it bothersome to deal with manually, so we won't use it here.

Let's see how this looks like in the Dog::ctor pseudocode:

Dog *__thiscall Dog::ctor(Dog *this)
{
  j_Animal::ctor((Animal *)this);
  j_Killable::ctor((Killable *)&this->__vftable_Killable);
  this->__vftable = (Dog_vftable *)&Dog::`vftable';
  this->__vftable_Killable = (Killable_vftable *)&Dog::`vftable';
  return this;
}

As typical for constructors, the base class constructors are called first. Then, the vftables required for Dog are set. But wait, something doesn't make sense: Why is Dog::vftable assigned to our __vftable_Killable? Well, I used IDA 7's naming here, and what I previously mentioned is that it doesn't mark which vftable maps to which base class anymore (unlike the script). With the IDA 6 script, it would say something like:

Dog *__thiscall Dog::ctor(Dog *this)
{
  j_Animal::ctor((Animal *)this);
  j_Killable::ctor((Killable *)&this->__vftable_Killable);
  this->__vftable = (Dog_vftable *)&Dog::`vftable for Animal';
  this->__vftable_Killable = (Killable_vftable *)&Dog::`vftable for Killable';
  return this;
}

Now the names aren't the same and make more sense, so don't get busted by this IDA 7 annoyance, double click the names to check where they'll actually lead.

I do not recommend setting the actual type of the vftables to their locations / names: You get nothing out of it, it just makes the disassembly output messy.


Bonus: Classes inherited from classes using multiple inheritance (wew)

This one kept me a little baffled for some time, maybe because it's not covered in most tutorials on the net on which I so rely on, or maybe because I'm just dumb. Let's inherit from Dog with our Terrier class.

class Terrier : public Dog {
    int _annoyanceLevel;
    virtual void setAge(int value) { _age = value; }
    virtual void makeSound() { cout << "Bark Bark not Woof Woof"; }
    virtual void annoy() { _annoyanceLevel++; }
};

MSVC layout: It doesn't seem too special. The Animal vftable still merges the new virtual methods of our Terrier, and everything else has its separate vftable:

class Terrier   size(24):
        +---
 0      | +--- (base class Dog)
 0      | | +--- (base class Animal)
 0      | | | {vfptr}
 4      | | | _age
        | | +---
 8      | | +--- (base class Killable)
 8      | | | {vfptr}
12      | | | _isDead
        | | | <alignment member> (size=3)
        | | +---
16      | | _furColor
        | +---
20      | _annoyanceLevel
        +---

Terrier::$vftable@Animal@:
        | &Terrier_meta
        |  0
 0      | &Terrier::setAge
 1      | &Terrier::makeSound
 2      | &Dog::setFurColor
 3      | &Terrier::annoy // Animal even takes the Terrier methods (greedy!)

Terrier::$vftable@Killable@:
        | -8
 0      | &Killable::kill
 1      | &Dog::makeDeathSound

IDA representation: It's pretty similar to our first Dog struct creation, just that we need to respect the second base class of Dog aswell.

struct Terrier_vtbl : Dog_vtbl {
  void (__thiscall *annoy)(Terrier *this);
};
struct Terrier_mbrs : Dog_mbrs {
  int _annoyanceLevel;
};
struct Terrier {
  Terrier_vtbl *__vftable;
  Animal_mbrs __members_Animal;
  Killable_vtbl *__vftable_Killable;
  Killable_mbrs __members_Killable;
  Terrier_mbrs __members;
};
  • Is "__thiscall" necessary to work? – savram Oct 30 '17 at 12:01
  • Yes, otherwise the calling convention to class methods and the decompiler output is ill-formed. (comment put into answer). – Ray Koopa Dec 17 '18 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.