Please bear with me as I explain this situation.
Maybe 10 days ago I took up the challenge of reverse engineering an Android app. Learning from scratch, I installed ADB, Apktool, Android Studio, Notepad++ with Smali highlighting etc. My approach has been to write test programs in Android studio that mirror the workings of the APK, and then decompile with Apktool to help me do Smali modifications. I've been successful in modifying the application to log all HTTP requests, headers, cookies, and post data to the android log.
My next challenge was to figure out how an important algorithm in the app works. This is what stumped me. I spent the last 3 or 4 days spending the majority of my day analysing Smali code making almost no progress. Apparently the algorithm is done (at least in part if not mostly) in a native library with the .so extension.
One of the extremely frustrating things about Reverse engineering is how small the community is. There are very few resources on the web (At least compared to other things). I've probably bitten off more than I can chew. I always attempt difficult projects that are above my skill level. For this task, I'm guessing I need to become very familiar with ARM and I'll have to use IDA Pro to analyse the .so file? To explain my knowledge level:
- I have very little experience using Ollydbg in Windows (I have slight understanding of registers and CMP, JMP, ADD commands, etc.)
- I have no experience using IDA Pro. I'm quite new to reverse engineering, but I've had success with simple Smali modding because it's simple in some situations.
- I know basic Java (To be clear, I'm not a complete beginner, variables, for loops, arrays, classes (to an extent) are second nature to me)
- I know basic C++ if not more than basic
- I've done a lot of VB.NET programming.
- I feel quite knowledgeable in Python.
So I'm experienced with programming, but not really reverse engineering. How in over my head am I attempting to understand this complex native library (When I already know practically nothing about native libraries/JNI)? Could the professionals here please give me some specific examples of how I can get to the level where I am knowledgeable enough to complete my goal? I don't want to just give up because this is a difficult challenge. Please give me suggestions of how I can progress enough to complete my goal. I assume I'll need to learn IDA Pro and how ARM works.