How does the sysenter instruction work? What instruction is called after sysenter? Is the next instruction saved in a register?

mov     edx,esp
  • it goes into ring 0 typically with edx pointing to the parameters and eax being the 'id' of what to do
    – evlncrn8
    Commented Sep 23, 2014 at 6:51
  • possible duplicate of How does sysenter work under Linux?
    – perror
    Commented Sep 23, 2014 at 7:02
  • I am suspecting that sysenter works the same for Linux and Windows... But, I didn't try with Windows, so you need to try out.
    – perror
    Commented Sep 23, 2014 at 7:04
  • @perror: the linked question explains how to use sysenter to call Linux syscalls. It does not explain how systenter works on the CPU level.
    – Igor Skochinsky
    Commented Sep 23, 2014 at 10:27
  • @Igor : Right, I probably misunderstood the question. Sorry.
    – perror
    Commented Sep 23, 2014 at 11:11

2 Answers 2


Here is a perfect explanation how sysenter works: http://www.osronline.com/article.cfm?id=257

In essence: All native API calls from User Mode have a body that simply loads an index into EAX, executes SystemCallStub, and returns

SystemCallStub saves a pointer to the top of the User Mode stack into EDX and executes a SYSENTER instruction

SYSENTER disables interrupts, switches the thread into Kernel Mode and executes the instruction located in the SYSENTER_EIP_MSR (which on XP SP1 is KiFastCallEntry)

KiFastCallEntry builds a trap frame so it knows where to go when returning back to User Mode, enables interrupts, and jumps into KiSystemService

KiSystemService, amongst doing other things, copies the parameters from the User stack (pointed to by EDX) and takes the value previously stored in EAX and executes the function located at KiServiceTable[EAX]

The native API now executes in Kernel Mode with the previous mode of the thread set to User Mode

  • exactly can you say where save address eiP after call sysenter in returning to user mode?in stack or register? Commented Sep 24, 2014 at 12:16
  • a trap frame is used and setup for EIP after returning to user-mode. read about how trap frames work exactly
    – pHiL
    Commented Sep 24, 2014 at 12:33
  • Did you mean Nowhere saving? can you show me a reference about trap frames work? tanx Commented Sep 24, 2014 at 12:49
  • the trap frame is set in kernel-mode so you cant modify it from user-mode. If you want to learn more, search for "interrupts, traps, exceptions"
    – pHiL
    Commented Sep 25, 2014 at 9:38

The SYSENTER instruction affects many registers, but most importantly it sets EIP to the value of the SYSENTER_EIP_MSR register (in IA-32 terminology). At boot-time, Windows sets SYSENTER_EIP_MSR to the address of ntoskrnl!KiFastCallEntry().

So at a high-level, SYSENTER "jumps" to ntoskrnl!KiFastCallEntry().

For more details, see http://www.codeguru.com/cpp/misc/misc/system/article.php/c8223/%22System-Call-Optimization-with-the-SYSENTER-Instruction.htm and http://trapframe.org/just-enough-kernel-to-get-by-2/.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.