How does sysenter instruction in windows 7 work?

Question

How does the sysenter instruction work? What instruction is called after sysenter? Is the next instruction saved in a register?

mov     edx,esp
sysenter

Answer 1

Here is a perfect explanation how sysenter works: http://www.osronline.com/article.cfm?id=257

In essence: All native API calls from User Mode have a body that simply loads an index into EAX, executes SystemCallStub, and returns

SystemCallStub saves a pointer to the top of the User Mode stack into EDX and executes a SYSENTER instruction

SYSENTER disables interrupts, switches the thread into Kernel Mode and executes the instruction located in the SYSENTER_EIP_MSR (which on XP SP1 is KiFastCallEntry)

KiFastCallEntry builds a trap frame so it knows where to go when returning back to User Mode, enables interrupts, and jumps into KiSystemService

KiSystemService, amongst doing other things, copies the parameters from the User stack (pointed to by EDX) and takes the value previously stored in EAX and executes the function located at KiServiceTable[EAX]

The native API now executes in Kernel Mode with the previous mode of the thread set to User Mode

Answer 2

The SYSENTER instruction affects many registers, but most importantly it sets EIP to the value of the SYSENTER_EIP_MSR register (in IA-32 terminology). At boot-time, Windows sets SYSENTER_EIP_MSR to the address of ntoskrnl!KiFastCallEntry().

So at a high-level, SYSENTER "jumps" to ntoskrnl!KiFastCallEntry().

For more details, see http://www.codeguru.com/cpp/misc/misc/system/article.php/c8223/%22System-Call-Optimization-with-the-SYSENTER-Instruction.htm and http://trapframe.org/just-enough-kernel-to-get-by-2/.