reverse engineering methodology

Question

I'm new to reverse engineering and I'm following the on line course provided by the opensecurity.info site.
The author provides a methodology for reverse engineering. He recommends to find a function of interest and pseudo code it.
From the wording used, it sounds like I should reverse the whole function from the its start to its end.
Is this really the recommended practice?
If my question isn't clear enough, here an hypothetical case: Let's say I want to understand the communication protocol used to exchange data between a malware and the command and control server. Let's say I found the recv call and this call is in the middle of an unknown function.
Should I really reverse this unknown function from its start or should I focus on the area of the code inside this function that calls "recv" and go from there?

Answer 1

Should I really reverse this unknown function from its start

Disclaimer: This is just my opinion. That said, my answer: no, don't do that unless you really need to do that. One of the first things I always say to my students when I teach reverse engineering is actually that: never try to reverse engineer a function line-by-line unless you really need to do so (and my experience says that most of the time you don't need to do so).

My recommendation is a bit different:

• Find "artifacts" than can give you an abstract idea about what a function does: called/referenced function names (i.e., APIs), constants, etc...
• After you have an abstract idea of what the function does, determine if you need to get more information from it.
• If so, try to determine grouping basic blocks but without analysing line-by-line, what a set of basic blocks do, trying to focus on what you do not understand or what you need to answer.
• If you still need more details or something is still unclear, do it basic block per basic block following the flow from the point that is not clear or from the point that you start to care.
• If, still, you need more information or it's still unclear, do it instruction by instruction trying to focus specifically on what you do not understand or what you need to answer and, only if still you cannot answer your questions, resort to reverse the whole function line-by-line.

Trying to go instruction by instruction or reversing the whole function when we really don't need to do so, it's easier to get lost/confused because we will be focusing on what some obscure instruction does or which flags it changes instead of focusing in answering the real questions. However, it will largely depend on why are you reverse engineering that thing. Are you trying to mimic/copy/extend an algorithm (i.e., a decryption algorithm) or recovering the source code for a binary? Then, yes, you will need to go instruction by instruction. Are you reverse engineering a piece of software/function in order to determine what it does? In general, no, you don't need to go instruction by instruction.

My 2 cents.

Answer 2

I suggest you took a look at the rest code, too. Finding something in specific can definitely be a breakthrough, but one line of code - especially in assembly - can't do a lot on its own. Look at the example you gave. You found how your malware communicates. Is that enough? Probably not. What is the nature of this communication? Is it a reverse shell? Is it used to transfer files? There are many different questions to be answered and - trust me - you will be very lucky if you can find them all at the same point of the code. Always look at the code around the area of interest you have. You might find functions that call other functions that do interesting stuff using the communication channel. One thing brings the other and at the end of the day, it will bring you the full structure and functionality of the malware.

P.S. Always use breakpoints when going from function to function.

Happy hunting!