Writing an hypervisor
The best way to currently do this is by creating a micro-hypervisor, exploiting technologies originally created to support faster virtualization to monitor usermode as well as kernelmode code. This became quite a common replacement for AVs once
SSDT hooking was protected by Microsoft using patch-guard. Additionally, a few security companies sprung out, using hypervisors and micro-kernels to protect and separate different components of the OS (an example is Bromium), and Microsoft recently announced their Application Guard which uses the same technique and actually installs an hypervisor when windows is installed.
Although setting up an hypervisor might seem like a silver bullet, it has many disadvantages:
- Although most CPUs today support hypervisors, not all do.
- Windows does not yet support nested-virtualization and is installing a hypervisor. Nexted virtualization has multiple complications and makes the hypervisor development process more complex. There can only be one hypervisor unless the first one installed supports nesting.
- Virtualization software heavily relies on virtualization support by the CPU, and running a VM and a hypervisored host might become tricky. Same goes for running a hypervisor inside a virtual machine (so you can debug it properly). This is a decent explanation of how to do that in VMWare.
Today, there are several open source hypervisors available, most for experimentation purposes and don't include a lot of actual features past the infrastructure needed to set up a functional hypervisor.
Here are some of those open source proof of concepts, that could be used as a starting point:
Hooking the Interrupt Vector Table, Interrupt Descriptor Table or the System Service Dispatch Table was very common in the not so distant past. Although this does not give the same granularity a hypervisor has and you won't be able to detect specific instructions, it was common practice that was partially replaced by hypervisors to get low level monitoring not provided by the OS.
The basic idea behind it is to override the list of addresses a CPU executes when certain interrupts are triggered. This is usually the OS's responsibility to handle interrupts and how many low level functionality is provided by the OS (such as managing memory and permissions, paging, multiple task support, input/output with peripherals etcetera). This technique was made somewhat irrelevant on certain OSes using techniques like PatchGuard.
rtdsc instruction is a bit special in a way not may other instructions are. Quoted from here:
When in protected or virtual 8086 mode, the time stamp disable (TSD) flag in register CR4 restricts the use of the RDTSC instruction as follows. When the TSD flag is clear, the RDTSC instruction can be executed at any privilege level; when the flag is set, the instruction can only be executed at privilege level 0.
It is special because it has it's own disabling control bit("time stamp disable") in
CR4. When that control bit is set, only kernelmode code is allowed to execute the
rdtsc instruction, but what's more interesting is that when this control bit is set, using
rdtsc from usermode will trigger a General Protection Fault, thus making such instructions detectable by the kernel.