How to find where the code is cycling - infinite loop?

Question

Is there some way or technique to determine which code cycles in the program? Like if my application gets on an infinite loop and I would like to know exactly where it loops?

Answer 1

Answering here in case other folks find this question and need a more detailed answer.

A couple of assumptions up front. OP said "application", so I'm assuming an OS that 1) has kernel/user mode, 2) is not a closed system (ie not an appliance or device), 3) has some notion of "process", and 4) has a debugger that allows you to debug a specific process (either an existing process or one the debugger launches). Think Windows, Linux.

For the sake of simplicity let's assume the app is single threaded. After breaking into the debugger a processor will be executing in the infinite loop (with some caveats).

(If the app is multithreaded then you need to switch to the context of each thread currently running and determine if an infinite loop is being executed by that thread. While unlikely, it is possible for an app to have multiple simultaneous infinite loops. Its also possible to have an infinite loop in a callback function in a thread the app didn't create.)

In many cases an infinite loop is just a few instructions (for the sake of discussion say < 50).

With a small infinite loop you can just single step (stepping over function calls) through the loop and characterize the whole loop. If the loop is more than say 8 instructions, it might help if you write down each chunk of assembly that has a single entry point and a single exit (a so called basic block) on a piece of paper and draw arrows between the basic blocks that represent all the jumps (and returns). Any arrows leading to code that doesn't get executed when you step through the loop are potential loop exits.

If there are no such arrows then the code was written to be an infinite loop - the loop may be intentional or the coder just made a coding error. If there are arrows out then they will (usually) be preceded by some sort of test/compare instructions. You need to investigate why those test/compare instruction never result in the loop being exited.

If the case that you've got a "big" infinite loop, stepping through each instruction becomes impractical (or at least tiresome). To find the reason for the loop you'll need to understand the code involved well enough to understand what the infinite loop is trying to accomplish and why the loop exit condition(s) is not being met. Here you'll use other RE techniques to understand the code making up the loop. Once you have that level of understanding you can focus on understanding why the loop exit conditions aren't being met.