Is it possible to use a hypervisor to trap all ring0<->ring3 transitions (both 0->3 and 3->0)?
Yes, it is possible. I do not want to go into details as there are a lot of comprehensive material available online:
- Intels manuals which you will have to study in any case if you really want to understand how things work (from personal experience)
- A VMM-based System Call Interposition Framework for Program Monitoring - one of the countless examples about sys call monitoring mainly for security purposes.
- HyperPlatform - Intel VT-x based hypervisor aiming to provide a thin VM-exit filtering platform on Windows.
- DdiMon - Monitoring and controlling kernel API calls with stealth hook using EPT
One of the techniques is to monitor execution of the kernel-mode entering code pointed by