0

I have an executable (.exe) containing some classes and functions definition and plan on using those functions from within a DLL, although I know to call a regular C style function using calling convention such as __stdcall e.g.

typedef int(*__stdcall CalleeType)(...);

int Caller(...)
{
    CalleeType pCallee = nullptr;
    pCallee = reintepret_cast<CalleeType>(BaseAddress + RVA);
    return pCallee(); //calls the function as expected from within the dll
}

Now my problem starts when I am trying to call functions in which ecx register is set before the call mostly a method used by _cdecl calling convention in c++. And if this ecx register is not set before the call of the function the application crashes.

I found a solution here to load a using Loadlibrary, it is for me to patch the entry point and set it to return 1, and other stuff that needs to be done before the EXE can become usable as a DLL although this method is a hack but it does show promising results in helping me load this program into memory and call those functions as expected. Although I am confused as to how I can be able to make use of those functions passing ecx since those are c++ style function calls.

My Questions are

  1. How do I make this program to execute after loading it using loadlibrary, so that the program can be able to initialize itself properly and can now be able to use properly?
  2. How do I make use of a function requiring that ecx register is set to the this pointer?

Any ideas or hint will be very useful to me thanks

4
  • In MSVC x86 __thiscall puts this into ecx – morsisko Sep 30 '20 at 20:26
  • I will give it a try, thanks – Dexter's Oct 1 '20 at 1:27
  • But does this also apply with x64? – Dexter's Oct 1 '20 at 1:28
  • @Dexter's. In Microsoft 64-bit calling convention, C++ this is passed in RCX. – fpmurphy Oct 1 '20 at 14:48
1

josh's answer is a good one, but one thing to note is that there is an alternative scheme that (usually) does not require inline assembly, and which handles the virtual functions case also. This is how I prefer to interact with C++ programs when I'm doing DLL injection/code reuse like your question is asking.

Begin by defining something for the class whose methods you want to call. If you know the full class declaration, recreate it in your calling program. If you know less than that -- say, you only know the size of the class -- you can do something like:

class MyClass {
  public:
  char mData[0x80]; // sizeof(MyClass) = 0x80
};

(If you don't know the size of the class, pick a large number in place of 0x80 and hope for the best. If the program crashes, pick a larger number and try again. Really though, try to figure out the size of the class, at minimum.)

Now, we can use member function pointers to force the compiler to set ecx (the this) pointer for us during the function pointer call. Namely:

typedef void (MyClass::*FuncToCall)(int arg1, void *arg2);

Similarly to josh's answer, now we want to cast the raw address of the member function in the DLL to a member function pointer of that type, and then call it:

#define FUNC_OFFSET 0x4320
FuncToCall f = (FuncToCall)((byte*)pDLL + FUNC_OFFSET);
MyClass *m = new MyClass();
m->*f(123, NULL); // member function pointer invocation syntax

EDITED TO NOTE: in my original answer, I'd forgotten to note that C++ is not as liberal about accepting casts of arbitrary types to member function pointers as C is about casting to ordinary function pointers. That is to say, the code in the below won't work directly; you'll need to basically twist the compiler's arm to make it work, but it is doable. See this StackExchange discussion of the situation. One of the answers proposes using a union, which is what I've typically done. See these two blog entries by Raymond Chen discussing why C++ is less tolerant about casts involving member function pointers (TL;DR multiple inheritance): one, two. Finally, here's an example of how I used these techniques in a recent project.

ORIGINAL TEXT CONTINUES HERE: You can even wrap this up into a proxy stub function:

#define FUNC1_OFFSET 0x4320

// Global function pointer(s)
// Note: these are GLOBAL variables, not class members
FuncToCall gfp_MemFunc1;

// Relocate all necessary addresses
// Call this once before calling member function pointers
void Init(byte *pDLL) {
  gfp_MemFunc1 = (FuncToCall)(pDLL + FUNC1_OFFSET);
}

class MyClass {
  public:
  char mData[0x80];

  // Make stub functions for members you want to call
  // Does not affect the layout of the class
  void MemFunc1Proxy(int arg1, void *arg2) {
    this->*gfp_MemFunc1(arg1, arg2);
  };
};

int main() {
  MyClass m();
  Init(/* address of DLL */);

  // Call member functions easily via stubs
  m.MemFunc1Proxy(123,NULL);
}

For virtual functions, there are two cases. In both cases, you'll need to add compatible virtual function delcarations (in the proper order) to the mock class declaration. Something like:

class MyClass {
  public:
  char mData[0x80];
  virtual void ~MyClass();
  virtual int VFunc0x4(int);
};

Now, as for the two different cases. First case: if you're able to call a function in the DLL to obtain an instantiated instance of your object, that's all you need; call a virtual function as follows:

MyClass *p = fpDLLFuncThatAllocatesMyClass();
int x = p->VFunc0x4(1);

If it's not easy or feasible to obtain a memory instance for MyClass from the DLL, you can just install its VTable yourself.

#define VTABLE_OFFSET 0x5670
MyClass *m = new MyClass();
void *VTableAddr = (byte*)pDLL + VTABLE_OFFSET;
// Set the VTable pointer to be the VTable address in the DLL
*reinterpret_cast<void *>(m) = VTableAddr;
int x = p->VFunc0x4(1);

This scheme gives you a bit more flexibility, and does not usually require inline assembly. The exception to this happens when the non-virtual member function that you want to call uses a non-standard calling convention, as happens frequently in MSVC-compiled x86. If the compiler determines that the function will not be accessed outside of the DLL, it's under no obligation to use plain __thiscall as the calling convention, and may use alternative register argument locations. In this situation, you'll have no choice but to use inline assembly language as in josh's answer above.

4
  • Rolf, the primary intention of inline assembly in my answer was to show the role of the ecx register by explicitly assigning it. In your answer the ecx assignment is hidden in the compiler code generation. The virtual functions calls are - as I stated - an additional complication, but of course useful to have demonstrated it. – josh Oct 1 '20 at 20:29
  • Yep, your answer was a good one and there's nothing wrong with it. Once I started using these techniques regularly in DLL injection/code reuse, I found that the kind of techniques I demonstrated in my answer were useful in practice to keep my code clean. Our answers largely focused on different aspects -- yours, background and demonstration; mine, using the techniques efficiently at scale. – Rolf Rolles Oct 1 '20 at 20:34
  • Very much agree! – josh Oct 1 '20 at 20:49
  • @RolfRolles is right, his idea does present a clean code indeed and does work. Thanks a lot – Dexter's Oct 1 '20 at 21:27
1

This is an interesting problem where some issues have to be solved.

For the first part of your question: How to run an exe from memory, you might want to consult the following link: https://stackoverflow.com/questions/305203/createprocess-from-memory-buffer

In the following, I will try to show you how to tackle your second question, how to call a class method of that exe being present in memory.

The following example (keep it as simple as possible) does not assume a DLL as a caller, but another exe. The mechanics should be exactly the same, however.

Part 1 : Called exe

Let us start with a very simple C++ Windows (VS2013) exe serving as the one to be called from another exe (or DLL in your case). I named this exe "ExeToBeCalled".

#include <iostream>

class myClass
{
public: 
    int myVar;
    void myFunc(int a, int b)
    {
        myVar = a - b;
    }
};

int main(int argc, char* argv[])
{
    myClass* pmc = new myClass();  //parameterless constructor
    myClass* pmc2 = new myClass(); //parameterless constructor
    pmc->myFunc(123, 23);
    pmc2->myFunc(456, pmc->myVar);
    std::cout << pmc2->myVar;
    return 0;
}

It contains a class with one method, called myFunc, which subtracts its two parameters and stores it into a public member variable, called myVar. The subtraction instead of e.g. an addition has been chosen to be easily able to decide whether the correct parameter calling sequence is used.

Two instances of this class are created, and the result is output to the console. Sloppy programming, as the delete has been omitted.

The interesting part is the fact that the class instances (created with "new" on the heap) contain only the myVar on the heap, because each instance must have its own myVar, whereas the myFunc is common for all instances and as a consequence its address is statically assigned by the compiler. This address has to be found out, e.g. by inspection in Ida. From your posting I assumed that you solved this problem already. The problem becomes more difficult for virtual methods, not considered in our simple example.

Part 2 : Caller exe

This of course is the interesting part. I also created an example (VS2013), like so:

#include <Windows.h> // necessary for the "LoadLibrary"
#include <iostream>

// class method, statically assigned by the compiler
// The address has to be found out, e.g. by inspection in Ida
// You solved this part of the problem already.
// As we call the function from assembler, the __cdecl is irrelevant.
// It would matter though if we called from C++.
// Note that the callee needs a valid ecx for the function to succeed.
typedef void(*__cdecl pmyFunc)(int a, int b);

    // assumed myClass layout, being a guess of the real layout of the class.
    // We only need the member variable
    // This "class" is for problem illustration only, we do NOT need it in our program.
    class myClass
    {
    public:
        int myResult;
    };

int main(int argc, char* argv[])
{
    HANDLE pExe = LoadLibrary("ExeToBeCalled.exe");

    // Address e.g. obtained by investigation of the callee in Ida.
    pmyFunc myFunc = (pmyFunc)((byte*)pExe + 0x4320); 

    // let us get some memory from the heap for the class
    // As we do not know the "real" layout of the class, we allocate one kB, hoping it will be enough
    // We do NOT refer to the "myClass" above
    void* pmyClass = new byte[1000];

    // Set it all to zero.
    for (int i = 0; i < 1000; i++)
        ((byte*)pmyClass)[i] = 0;

    // let us switch to assembler for the call
    _asm
    {
        // We have to provide a valid memory in ecx where the real myclass can store its result
        // If we provide here for testing a zero into ecx, the app crashes.
        mov ecx, pmyClass;
        push 333;
        push 222;
        call myFunc; //result should be 222-333 = -111
    }
    // By inspection of the pmyClass memory in the debugger after the call to myFunc,
    //    we saw that the real myClass just stored the result into the first four bytes.
    // Of course, in reality this is not known beforehand, thus - in a real situation -
    //    the whole 1kB could have been searched for the expected result, determining
    //    the offset of myResult from the ecx value in reality.
    int result = *(((int*)pmyClass));
    std::cout << result;
    return 0;
}

It has to be noted that our test exe from part 1 has to be compiled in Debug instead of Release mode. It is too simple for the compiler which optimizes it and does not even bother to call the myFunc, but directly calculates the result.

To summarize, for the solution of your problem the following must be assumed or known:

  1. The address of the class method. We assumed a static address which need not to be the case in general. "Static Address" here means a constant offset (RVA) of the exe's start address obtained by LoadLibrary, which usually differs on every load (ASLR).

  2. The way the compiler of the exe used to call a method of a class. Here we assumed the passing of the "this" pointer in the ecx register.

  3. The memory location where the result of the method call is stored. This also can be arbitrarily complicated in general, in our demo the simplest possible memory layout turned out to be correct.

In one of the comments the question about 64Bit has been asked. Of course, there the situation might be different, the method to investigate the problem would be rather similar.

I hope this little example could enlighten one method how to attack your problem.

2
  • It does help, although I am still not clear on other forms or method I can use to set the ecx to this without having to pass it manually using inline assembly. – Dexter's Oct 1 '20 at 21:21
  • I am assuming there's a way around this, maybe a hack or something that makes it easy. What about using the vtable of the class? – Dexter's Oct 1 '20 at 21:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.