3

I have absolutely no experience in reversing real-world binary codes, so I wonder how the obfuscated codes prevent reversers. I doubt that the reverses always find some ways to understand what are hiden inside, even for heavy obfuscated codes, but I do not know how they think about them.

That partly comes from this question in detecting recursive calls where both two answers give a static approach: looking recursively in the functions called by the original function whether it is re-called.

In somehow theoretical manner, this approach can be bypassed if the programmer uses the continuation passing style, that is because there is no more explicit

call myself

inside the code. The following program I have implemented to test out this idea:

template<typename T> 
auto obf_if(bool p, T a, T b) -> T
{
  T* pts[4] = { &a, &b, &a + 1, &b + 1 };
  return *pts[int{ p }];
}

template<typename T>
auto obf_cmp(T a, T b) -> int
{
  return obf_if<int>(a == b, 0, obf_if<int>(a < b, -1, 1));
}

using obf_strcmp_t = std::function < int(char*, char*) >;

auto h_strcmp(obf_strcmp_t func, char* str1, char* str2) -> int
{
  return obf_if<int>((*str1 == *str2) && (*str1 != 0), 
                     func(str1 + 1, str2 + 1), obf_cmp<int>(*str1, *str2));
}
using h_strcmp_t = decltype(h_strcmp);

obf_strcmp_t y_strcmp(h_strcmp_t func)
{
  return std::bind(func, std::bind(y_strcmp, func), 
                   std::placeholders::_1, std::placeholders::_2);
}

int main(int argc, char* argv[]) 
{
  char str1[] = "ab"; 
  char str2[] = "ac";
  return y_strcmp(h_strcmp)(str1, str2);
}

This is a trivial implementation of strcmp using the y combinator. But this piece of codes leads to the fact that there is no more direct call inside the implementation (even no conditional jump), except the first one

y_strcmp(h_strcmp)(str1, str2)

As an amateur, I have even loaded the binary code (compiled by VS2013) in IDA and see a big mess where calls are replaced by

 call edx

However because I write it I know how to detect this (e.g. the implicit recursive calls are detected by tracing the arguments passed into the function, the value of edx can only be one of passed arguments), and I think so do the reversers. So my question is:

Suppose that you do not know this trick, does it prevent you in understanding the binary code?

NB Because w-s has suggested that this question is only an opinion-based one, so it will be closed sooner or later, but I very appreciate if someone gives an idea.

  • 2
    Strictly speaking it is opinion based question. Less strictly speaking there is dynamic analysis. Such kind of obfuscation (as almost any other obfuscation) will not prevent anyone to read this code because there is no obfuscation that can prevent reading the code at all, obfuscation just makes code reading slower. – w s May 18 '14 at 10:15
  • Thanks a lot w-s, I am very agree with you. That leads me to this question because when I read some very nice approachs in code obfuscation, the question in my head is always: if the reversers know this trick, does it prevent them anymore. – Ta Thanh Dinh May 18 '14 at 10:20
  • 1
    It is much more psychological question then question of reverse engineering. I know a lot of reversers that will stop reversing any code when they see any kind of obfucation but I also know some reverse engineers that will not recognize here any obfuscation and will define this code as "over-designed" and over C++ed :) – w s May 18 '14 at 10:40
  • I got your joke w-s :D. I think that I should delete this psycho question before it is down-voted by many people and I lose all of my reputation points. – Ta Thanh Dinh May 18 '14 at 11:16
  • 1
    This will only slow down a dedicated reverser, but because the code itself is not hiding its functionality in any way, it can be reversed given enough time. And like the answer below, there are ways to identify recursive functions, no matter how elusive the programmer may try to be. stack based traces are a good way to spot this. – gandolf May 27 '14 at 13:53
1

Hook the start of all functions that you're concerned might be recursive with EasyHook. Have your hooking function log the function address in the thread local storage (to avoid synchronisation issues) then look at either a full stack back trace or just the calling function. Recursive functions will be obvious.

  • Thanks a lot offbyseveral, I am very appreciated your answer. I am very agree with you that this obfuscation trick (which bypasses the static analysis) can be bypassed easily by the dynamic analysis (in particular, nothing is hidden inside a "session" of dynamic analysis). But the problem is the dynamic analysis is quite bad in detecting which might "potentially become" recursive calls. – Ta Thanh Dinh May 28 '14 at 8:56
  • 1
    To do what you want to do rigorously is impossible. Imagine a program that only becomes recursive when a structure is full of data which correctly decrypts using a key. Or which is based on a specific password. Static analysis isn't going to cut it for all but the most trivial programs. – offbyseveral May 28 '14 at 9:02
  • 1
    Imagine that a program does call edx based on data lookup in a vtable that has been modified by an attacker with an exploit to become recursive. To solve your problem rigorously you'd need to first show that the program state can't be modified in this way and then you'd be a very rich person indeed! – offbyseveral May 28 '14 at 9:06
  • Thanks a lot offbyseveral, you are absolutely correct. The question is not rigorous (coincidentally I have just used the word "rigorous" to reponse the comment of gandolf) so it is impossible to give a precise answer. I just want to receive your precious ideas. – Ta Thanh Dinh May 28 '14 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.