VMProtect Anti-debugging method (without WiNAPI)

Question

First, sorry for my bad english.

I'm trying to make VMProtect unpacker with unicorn emulator, but one of my sample shows me like this anti-debugging message:

WTSSendMessageW: "A debugger has been found running in your systeam. Please, unload it from memory and restart your program"

My sample is x86 pe dll and i don't know exact VMProtect version (but above 3.0 is definitely) In unpack process, WinAPI calling sequence looks like below:

IsWow64Process
IsDebuggerPresent (make return 0)
CheckRemoteDebuggerPresent (make pbDebuggerPresent as 0)
NtQueryInformationProcess (make ProcessInformation as 0, return STATUS_PROT_NOT_SET)
NtSetInformationThread 
NtQuerySystemInformation (3 times, maybe this is kernel anti-debugging routines)
GetModuleFileNameW 
NtOpenFile (for file corruption check?)
NtCreateSection
NtMapViewOfSection
NtUnmapViewOfSection
NtClose
NtProtectVirtualMemory
// I think below code is detection routine
GetModuleFileNameW
GetProcessWindowStation
GetUserObjectInformation
LoadLibraryA (wtsapi32.dll)
WTSSendMessageW (Anti-debugging detection message)

I think every detection routine has called before NtProtectVirtualMemory, and the rest of code is unpack their's code to text section with several NtProtectVirtualMemory functions. But, after the first NtProtectVirtualMemory has called, and samples shows detection routine without any suspicious WinAPI calling.

I have issue with unicorn emulator hook code, so i cannot logged all unpacking code, and when i running this sample with x86dbg, i cannot make same situation that i face on.

So, I wonder is there any anti-debugging technique which is not based on WinAPI? (single-step exception not occured yet, when detection timing)

I hope for your guide. thank you.

Answer 1

Yes, you are correct, there is a very common anti-emulation detection technique that is based on the rdtsc instruction, here are 2 variations of it (taken from al-khaser):

BOOL rdtsc_diff_locky()
{
    ULONGLONG tsc1;
    ULONGLONG tsc2;
    ULONGLONG tsc3;
    DWORD i = 0;

    // Try this 10 times in case of small fluctuations
    for (i = 0; i < 10; i++)
    {
        tsc1 = __rdtsc();

        // Waste some cycles - should be faster than CloseHandle on bare metal
        GetProcessHeap();

        tsc2 = __rdtsc();

        // Waste some cycles - slightly longer than GetProcessHeap() on bare metal
        CloseHandle(0);

        tsc3 = __rdtsc();

        // Did it take at least 10 times more CPU cycles to perform CloseHandle than it took to perform GetProcessHeap()?
        if ((LODWORD(tsc3) - LODWORD(tsc2)) / (LODWORD(tsc2) - LODWORD(tsc1)) >= 10)
            return FALSE;
    }

    // We consistently saw a small ratio of difference between GetProcessHeap and CloseHandle execution times
    // so we're probably in a VM!
    return TRUE;
}


/*
CPUID is an instruction which cauz a VM Exit to the VMM, 
this little overhead can show the presence of a hypervisor
*/

BOOL rdtsc_diff_vmexit()
{
    ULONGLONG tsc1 = 0;
    ULONGLONG tsc2 = 0;
    ULONGLONG avg = 0;
    INT cpuInfo[4] = {};

    // Try this 10 times in case of small fluctuations
    for (INT i = 0; i < 10; i++)
    {
        tsc1 = __rdtsc();
        __cpuid(cpuInfo, 0);
        tsc2 = __rdtsc();

        // Get the delta of the two RDTSC
        avg += (tsc2 - tsc1);
    }

    // We repeated the process 10 times so we make sure our check is as much reliable as we can
    avg = avg / 10;
    return (avg < 1000 && avg > 0) ? FALSE : TRUE;
}

hope this helps.