I don't know exactly how this feature is implemented in Ollydbg, but we can have a look at how it is implemented in x64_dbg as it is open-source.
The Ctrl-F9 (execute till return) functionality is implemented by cbDebugRtr in x64_dbg_dbg/debugger_commands.cpp. We can see that it calls StepOver which is a method provided by TitanEngine.
CMDRESULT cbDebugRtr(int argc, char* argv[])
{
StepOver((void*)cbRtrStep);
cbDebugRun(argc, argv);
return STATUS_CONTINUE;
}
In the TitanEngine document the description of StepOver is described as follows :
StepOver function
The StepOver traces code by single-stepping over calls. This tracing
function sets an INT3 breakpoint after the call, which is used to call
your callback. There is no guarantee that code execution will return
from that call, and thus no guarantee that your callback will ever be
called. The breakpoint is run only once, and is removed once your
callback has finished.
TitanEngine is also open-source, here is the implementation of StepOver in TitanEngine / TitanEngine.Debugger.Control.cpp
__declspec(dllexport) void TITCALL StepOver(LPVOID StepCallBack)
{
ULONG_PTR ueCurrentPosition = GetContextData(UE_CIP);
unsigned char instr[16];
MemoryReadSafe(dbgProcessInformation.hProcess, (void*)ueCurrentPosition, instr, sizeof(instr), 0);
char* DisassembledString = (char*)StaticDisassembleEx(ueCurrentPosition, (LPVOID)instr);
if(strstr(DisassembledString, "CALL") || strstr(DisassembledString, "REP") || strstr(DisassembledString, "PUSHF"))
{
ueCurrentPosition += StaticLengthDisassemble((void*)instr);
SetBPX(ueCurrentPosition, UE_BREAKPOINT_TYPE_INT3 + UE_SINGLESHOOT, StepCallBack);
}
else
StepInto(StepCallBack);
}
After each step the callback function cbRtrStep is called which checks if the current instruction is a RET and if so the debugger is stopped. (x64_dbg_dbg/debugger.cpp)
void cbRtrStep()
{
unsigned int cipch = getCIPch();
if(cipch == 0xC3 or cipch == 0xC2)
cbRtrFinalStep();
else
StepOver((void*)cbRtrStep);
}
So, to sum up, Ctrl-F9 is doing step-over until it encounters a RET instruction. If you press F7 and Ctrl-F9 the debugger will single-step the instructions of the function; the function you are calling might be doing debugger detection that is not triggered when setting a breakpoint after this function.
