Hooking functions in a VMProtect'ed executable

Question

I'm trying to solve the Sharif CTF 2016 HI2.exe challenge. This challenge requires a specific (impossible to reach) environment (64 processors, 128GB RAM, ...) to display the flag. As the executable is packed with VMProtect, the best alternative seems to hook functions responsible for the environment specification checks, and modify their return values at run time. I'm using OllyDbg with the StrongOD plugin. I have documented all details here: https://www.aldeid.com/wiki/SharifCTF-2016/hi.

I have achieved the first step (I have modified values of the SYSTEM_INFO structure manipulated by the GetSystemInfo function) but failed to reach my second breakpoint, set on the GlobalMemoryStatusEx function. The program displays the following error message before my 2nd BP is reached: A debugger has been found running on your system.

I have tried to play with a few options from the StrongOD plugin (HidePEB, KernelMode, SkipSomeExceptions) but it either doesn't change anything or crashes the executable before the 2nd BP.

How can I achieve this goal? How should I configure OllyDbg and/or StronOD to prevent the program from knowing it's being debugged?

Many thanks in advance for your help.

Answer 1

StrongOD can have unexpected behaviour if you are using Win7 as your debugging environment. Try to use ScyllaHide plugin with the VMProtect profile on, ignore Invalid or privileged instruction exception in the debugging options.

Answer 2

If there's no way to debug it, use system mechanisms to load your DLL into protected process, then do whatever you like.