Detecting loops from binary code isn't easily done. There are many algorithms based on multiple graph data structures (CFG, SSA form, DDG, ...).
I have provided a set of additional helpful documents link1, link2, link3.
You can check MAQAO's implementation of an SSA based loop detector. Also, I would recommend you checking DynInst's code and documentation link4.
Now, you can implement an assembly code analyzer that can detect basic blocks/loops by following branches.
- Disassemble your binary and go thhrough instructions.
- If a conditional jump is encountered (Jxx pattern), check whether it jumps above or below.
If the jump points above, check the distance (Delta(BRANCH_ADDRESS, BRANCH_TARGET_ADDRESS)), if the distance isn't too large (you'll have to decide of a limit) then you can apply some analysis on the branch condition (check whether it contains a register which is displaced [increment/decrement]). The branch condition analysis could be a simple pattern matching :
If it all checks out, you got yourself a basic block which has a high probability of being a loop.
- If the jump points below, its probability of being part of a loop control structure is less likely. You can therefore ignore it and move on to the next instruction.
If you need more hairy details, I'll be glad to provide more ;)