How Do I get proficient at Reverse engineering?

Question

Please bear with me as I explain this situation.

Maybe 10 days ago I took up the challenge of reverse engineering an Android app. Learning from scratch, I installed ADB, Apktool, Android Studio, Notepad++ with Smali highlighting etc. My approach has been to write test programs in Android studio that mirror the workings of the APK, and then decompile with Apktool to help me do Smali modifications. I've been successful in modifying the application to log all HTTP requests, headers, cookies, and post data to the android log.

My next challenge was to figure out how an important algorithm in the app works. This is what stumped me. I spent the last 3 or 4 days spending the majority of my day analysing Smali code making almost no progress. Apparently the algorithm is done (at least in part if not mostly) in a native library with the .so extension.

One of the extremely frustrating things about Reverse engineering is how small the community is. There are very few resources on the web (At least compared to other things). I've probably bitten off more than I can chew. I always attempt difficult projects that are above my skill level. For this task, I'm guessing I need to become very familiar with ARM and I'll have to use IDA Pro to analyse the .so file? To explain my knowledge level:

• I have very little experience using Ollydbg in Windows (I have slight understanding of registers and CMP, JMP, ADD commands, etc.)
• I have no experience using IDA Pro. I'm quite new to reverse engineering, but I've had success with simple Smali modding because it's simple in some situations.
• I know basic Java (To be clear, I'm not a complete beginner, variables, for loops, arrays, classes (to an extent) are second nature to me)
• I know basic C++ if not more than basic
• I've done a lot of VB.NET programming.
• I feel quite knowledgeable in Python.

So I'm experienced with programming, but not really reverse engineering. How in over my head am I attempting to understand this complex native library (When I already know practically nothing about native libraries/JNI)? Could the professionals here please give me some specific examples of how I can get to the level where I am knowledgeable enough to complete my goal? I don't want to just give up because this is a difficult challenge. Please give me suggestions of how I can progress enough to complete my goal. I assume I'll need to learn IDA Pro and how ARM works.

Thanks

Answer 1

I spent the last 3 or 4 days spending the majority of my day analysing Smali code making almost no progress.

I know it feels like you've made "no progress", but I'd encourage you to not look at it that way. You spent 3 or 4 days figuring out which approaches don't work, which is in itself progress. And you also built up 3 or 4 days of reverse engineering experience, regardless of how fruitful the immediate outcome was.

Apparently the algorithm is done (at least in part if not mostly) in a native library with the .so extension.

...

How in over my head am I attempting to understand this complex native library?

Given that you've determined that the Java code calls out to the .so library, it shouldn't be too difficult to find the native function in the .so library since the native function would be exported-by-name for JNI compatibility. This means that you don't in fact need to "understand this complex native library" in its entirety, but rather just the one native function in question (in addition to the functions that that function calls).

If the native code is not heavily obfuscated and if you have Hex-Rays for ARM, it should be relatively easy to understand the target function. If, on the other hand, you don't have Hex-Rays, then you can use the evaluation version of IDA Pro to disassemble the target function. You'd need to manually analyze the ARM instructions to determine what the function is doing. Although that can be tedious, ARM instructions and the architecture in general are very well-documented. Approach it as you would when learning any new programming language.

Start analyzing the function from the very first instruction, and keep high-level notes on what each instruction is doing in the context of the function. Keep track of what values get stored in what registers and how memory is used. On first-pass, your goal is to determine what the function is doing (based on the instructions). Once you've extrapolated what the function is doing, your second-pass should focus on trying to understand why the function is doing what it's doing. After some time, things should begin to "click" in your mind, you'll get that "aha!" moment, and you'll understand how the target algorithm works.