2

How does the sysenter instruction work? What instruction is called after sysenter? Is the next instruction saved in a register?

mov     edx,esp
sysenter
  • it goes into ring 0 typically with edx pointing to the parameters and eax being the 'id' of what to do – evlncrn8 Sep 23 '14 at 6:51
  • possible duplicate of How does sysenter work under Linux? – perror Sep 23 '14 at 7:02
  • I am suspecting that sysenter works the same for Linux and Windows... But, I didn't try with Windows, so you need to try out. – perror Sep 23 '14 at 7:04
  • @perror: the linked question explains how to use sysenter to call Linux syscalls. It does not explain how systenter works on the CPU level. – Igor Skochinsky Sep 23 '14 at 10:27
  • @Igor : Right, I probably misunderstood the question. Sorry. – perror Sep 23 '14 at 11:11
8

Here is a perfect explanation how sysenter works: http://www.osronline.com/article.cfm?id=257

In essence: All native API calls from User Mode have a body that simply loads an index into EAX, executes SystemCallStub, and returns

SystemCallStub saves a pointer to the top of the User Mode stack into EDX and executes a SYSENTER instruction

SYSENTER disables interrupts, switches the thread into Kernel Mode and executes the instruction located in the SYSENTER_EIP_MSR (which on XP SP1 is KiFastCallEntry)

KiFastCallEntry builds a trap frame so it knows where to go when returning back to User Mode, enables interrupts, and jumps into KiSystemService

KiSystemService, amongst doing other things, copies the parameters from the User stack (pointed to by EDX) and takes the value previously stored in EAX and executes the function located at KiServiceTable[EAX]

The native API now executes in Kernel Mode with the previous mode of the thread set to User Mode

  • exactly can you say where save address eiP after call sysenter in returning to user mode?in stack or register? – user2522767 Sep 24 '14 at 12:16
  • a trap frame is used and setup for EIP after returning to user-mode. read about how trap frames work exactly – pHiL Sep 24 '14 at 12:33
  • Did you mean Nowhere saving? can you show me a reference about trap frames work? tanx – user2522767 Sep 24 '14 at 12:49
  • the trap frame is set in kernel-mode so you cant modify it from user-mode. If you want to learn more, search for "interrupts, traps, exceptions" – pHiL Sep 25 '14 at 9:38
3

The SYSENTER instruction affects many registers, but most importantly it sets EIP to the value of the SYSENTER_EIP_MSR register (in IA-32 terminology). At boot-time, Windows sets SYSENTER_EIP_MSR to the address of ntoskrnl!KiFastCallEntry().

So at a high-level, SYSENTER "jumps" to ntoskrnl!KiFastCallEntry().

For more details, see http://www.codeguru.com/cpp/misc/misc/system/article.php/c8223/%22System-Call-Optimization-with-the-SYSENTER-Instruction.htm and http://trapframe.org/just-enough-kernel-to-get-by-2/.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.