Is it possible to hook CPU instructions without severe performance impact?

Question

I have been working on some unpackers, and I noticed that at their end, they mostly use this:

mov edx, [ebp+oep]
call edx ; now, you're at the OEP of the original binary

Or this:

mov ecx, [ebp+oep]
jmp ecx

Or this:

mov eax, [ebp+oep]
push eax
ret

Knowing that, with a quick hook on absolute jumps/calls you'd unpack stuff in literally seconds. Apart from that, some software uses cpuid and rdstc to identify you, while protecting the assembly heavily with a lot of obfuscation. Put an "instruction breakpoint" on either, and you just bypassed everything faster than your eye blinks!

Is what I'm saying possible, without severe performance hits? I am aware of tracing, but that's really really slow so it's not an option.

Edit: I am aware of the existence of hypervisors, I just have no idea if they fit my situation, and if so, I wouldn't know how to apply those techniques to work, so any pointers, even if Google keywords, would be great.

Answer 1

Writing an hypervisor

The best way to currently do this is by creating a micro-hypervisor, exploiting technologies originally created to support faster virtualization to monitor usermode as well as kernelmode code. This became quite a common replacement for AVs once SSDT hooking was protected by Microsoft using patch-guard. Additionally, a few security companies sprung out, using hypervisors and micro-kernels to protect and separate different components of the OS (an example is Bromium), and Microsoft recently announced their Application Guard which uses the same technique and actually installs an hypervisor when windows is installed.

Technical implementation

Although setting up an hypervisor might seem like a silver bullet, it has many disadvantages:

1. Although most CPUs today support hypervisors, not all do.
2. Windows does not yet support nested-virtualization and is installing a hypervisor. Nexted virtualization has multiple complications and makes the hypervisor development process more complex. There can only be one hypervisor unless the first one installed supports nesting.
3. Virtualization software heavily relies on virtualization support by the CPU, and running a VM and a hypervisored host might become tricky. Same goes for running a hypervisor inside a virtual machine (so you can debug it properly). This is a decent explanation of how to do that in VMWare.

Today, there are several open source hypervisors available, most for experimentation purposes and don't include a lot of actual features past the infrastructure needed to set up a functional hypervisor.

Here are some of those open source proof of concepts, that could be used as a starting point:

1. https://github.com/hzqst/Syscall-Monitor
2. https://github.com/ionescu007/SimpleVisor
3. https://github.com/asamy/ksm
4. https://github.com/tandasat/HyperPlatform

overwriting IVT/IDT/SSTD

Hooking the Interrupt Vector Table, Interrupt Descriptor Table or the System Service Dispatch Table was very common in the not so distant past. Although this does not give the same granularity a hypervisor has and you won't be able to detect specific instructions, it was common practice that was partially replaced by hypervisors to get low level monitoring not provided by the OS.

The basic idea behind it is to override the list of addresses a CPU executes when certain interrupts are triggered. This is usually the OS's responsibility to handle interrupts and how many low level functionality is provided by the OS (such as managing memory and permissions, paging, multiple task support, input/output with peripherals etcetera). This technique was made somewhat irrelevant on certain OSes using techniques like PatchGuard.

An rdtsc trick

The rtdsc instruction is a bit special in a way not may other instructions are. Quoted from here:

When in protected or virtual 8086 mode, the time stamp disable (TSD) flag in register CR4 restricts the use of the RDTSC instruction as follows. When the TSD flag is clear, the RDTSC instruction can be executed at any privilege level; when the flag is set, the instruction can only be executed at privilege level 0.

It is special because it has it's own disabling control bit("time stamp disable") in CR4. When that control bit is set, only kernelmode code is allowed to execute the rdtsc instruction, but what's more interesting is that when this control bit is set, using rdtsc from usermode will trigger a General Protection Fault, thus making such instructions detectable by the kernel.