What skills, other than practice, should I acquire to be good at reverse engineering softwares?

Question

What would help me to be improve my self, What subjects should I study and learn from which increase my ability to be be better at reverse engineering a software ? i.e: Calculus, Integration ?

Answer 1

Such questions rarely have straightforward answers but I'll try to point you on a right path.

Firstly, Build a toolbox : gather as much tools as you can & start playing. Then, master the ones which you find useful or develop your own when you can do better. By tools, I mean disassemblers, hexeditors, decompilers, debuggers, automation scripts, ...

Secondly, Be methodical : with experience & practice you'll be able to come up with a set of steps to follow when reversing. These steps will certainly have to be updated with more experience but they'll surely help you find valuable information & data about your target.

Thirdly, Never blindly trust your tools or Know your tools : always cross-reference the outputs of different tools (use multiple disassemblers, ...). Most tools have strengths and weaknesses and you must know them in order to understand what you're looking at. For example, objdump (the binutils disassembler) performs linear sweep, a technique which can easily be fooled by misalignments and mixing data with instructions. A wise reverser would use another disassembler which implements another disassembly algorithm (IDAPro, r2 - radare, ...) and compare the results.

Fourthly, RTFM : documentation will in most, if not all, cases be your best friend. Any documentation related to the architecture, the tools you're using, the target application, the programming language, the compiler, ... is a gold mine. It will help you understand how all the machinery manages to function. You'll have to educate yourself in compilation (use & study different compilers : GNU's gcc, Intel's icc, PGI's pgcc, CAPS compilers suite, IBM's xlc, ...), static & dynamic linking, object file formats (ELF32, ELF64, PE, COFF, Macho, ...), calling conventions and ABIs, compiler optimizations, link time optimizations, code morphing, cryptography, ... to know how the application was built, obfuscated, and secured. All that, in order to find an exploitable weak spot in the chain. I would also recommend you following the scientific work conducted in all those areas, especially obfuscation, static & dynamic analysis, instrumentation, binary code patching, disassembly, ... The field of binary code analysis is still at its infancy, and every now and then researchers come up with great techniques and algorithms which end up implemented in many industrial products.

Fifthly, Be sharp : there's no such thing as garbage. As Lavoisier once said : "Rien ne se perd, rien ne se crée, tout se transforme." ("Nothing is lost, nothing is created, everything is transformed."). Keep in mind that any piece of valuable information can help. It can be poetry, calculus, physics, Boolean algebra, ... you never know what you'll stumble upon. A successful reverse engineer is an educated one !

Finally, Develop top programming skills : : learn as many programming languages as you can : assembly (x86, ARM, 6502, ...), C, C++, Python, Haskell, ... and learn how to extract code patterns from low level constructs - this is just a matter of experience & sharpness. Each language is designed on top of a set of paradigms : Functional programming, OOP, ... Knowing those paradigms helps a lot when analyzing code.

I do not think there's enough space for me to cover everything but I hope this will help you narrow your search & find useful documentation, tutorials, ... on the web.

Answer 2

There's always some special kind of software that requires specific mathematical skills. For example, if you want to reverse some malware that encrypts files, skills in cryptography will help you identify algorithms. Or, if you want to reverse some software that calculates, say, antennas, knowing equations from radio engineering will help you.

But, to improve your skills in reversing "general" software, the best you can do is learn several assembly languages, i'd recommend at least intel (32 bit and 64 bit), and arm instruction sets, registers, and flags. Also, get at least some theoretical knowledge about compilers, how they work, and which approaches to compiler optimizations there are.

Write small (C) programs, compile them to assembler, use different optimization flags, and learn how the compiler translates them into assembler code. Learn about the various ways to arrange the stack, so you'll recognize patterns when you see compiled assembly code.

While, in some cases, you'll encounter hand-crafted assembly, most of what you're going to reverse in "real life" will be compiled in some way or other, and in most cases, it'll be C or C++ that gets compiled. (C# and Java as well, but these are special cases, because decompilation is easy with those except if obfuscated). See this question Is there a way to adjust local variables when a function doesn't utilize ebp?, or this Can someone tell me what this memcpy or maybe it's a memset? looks like a memset, rep movsd and rep movsb This is a packet sending function - both are easy to solve if you just know how compilers work, and if you've compared respective assembly/C code.

Apart from that, as you already said, lots of experience, in reversing AND in programming. Theres's lots of algorithms that are quite hard to understand when you just see the implementations - a few weeks ago, i came across a function that had lots of ifs and variable swapping that i wasn't able to make any sense of, until i suddenly realized that i used exactly the same stuff some 10 years ago when i once implemented Bresenham's algorithm. This is the kind of stuff you can't really learn - a lot of the "magical" realizations of experienced reversers is not magic, they're just good memory.