I don't know exactly how this feature is implemented in Ollydbg, but we can have a look at how it is implemented in x64_dbg as it is open-source.
The Ctrl-F9 (execute till return) functionality is implemented by
x64_dbg_dbg/debugger_commands.cpp. We can see that it calls
StepOver which is a method provided by TitanEngine.
CMDRESULT cbDebugRtr(int argc, char* argv)
In the TitanEngine document the description of StepOver is described as follows :
The StepOver traces code by single-stepping over calls. This tracing
function sets an INT3 breakpoint after the call, which is used to call
your callback. There is no guarantee that code execution will return
from that call, and thus no guarantee that your callback will ever be
called. The breakpoint is run only once, and is removed once your
callback has finished.
TitanEngine is also open-source, here is the implementation of
TitanEngine / TitanEngine.Debugger.Control.cpp
__declspec(dllexport) void TITCALL StepOver(LPVOID StepCallBack)
ULONG_PTR ueCurrentPosition = GetContextData(UE_CIP);
unsigned char instr;
MemoryReadSafe(dbgProcessInformation.hProcess, (void*)ueCurrentPosition, instr, sizeof(instr), 0);
char* DisassembledString = (char*)StaticDisassembleEx(ueCurrentPosition, (LPVOID)instr);
if(strstr(DisassembledString, "CALL") || strstr(DisassembledString, "REP") || strstr(DisassembledString, "PUSHF"))
ueCurrentPosition += StaticLengthDisassemble((void*)instr);
SetBPX(ueCurrentPosition, UE_BREAKPOINT_TYPE_INT3 + UE_SINGLESHOOT, StepCallBack);
After each step the callback function
cbRtrStep is called which checks if the current instruction is a
RET and if so the debugger is stopped. (
unsigned int cipch = getCIPch();
if(cipch == 0xC3 or cipch == 0xC2)
So, to sum up, Ctrl-F9 is doing step-over until it encounters a
RET instruction. If you press F7 and Ctrl-F9 the debugger will single-step the instructions of the function; the function you are calling might be doing debugger detection that is not triggered when setting a breakpoint after this function.