Struggling between syscall or sysenter ( Windows )

Question

As I understand, from windows XP, there is a sysenter instruction introduced, instead of int 2e interrupt.

But on win 10 there is syscall instruction used in ntdll. So is the sysenter used only for x86 windows systems? And syscall only for x64 bit systems?

Answer 1

The Intel and AMD instruction sets are similar but not identical.

There are many examples of that: FMA3/FMA4, AMD-V and VT-x (and their extensions), etc.

The fast system call interface is yet another difference.

• Intel supports SYSENTER in all modes¹ - Legacy Mode (or, on the few 32-bit only Intel CPUs, simply Protected Mode), Long Mode and Compatibility Mode. (It doesn't work in Real Mode, obviously, and the from now on I'm ignoring it.)

• Intel supports SYSCALL only in 64-bit Long Mode² (not compat. mode). It also requires setting a bit in some MSR.

• AMD supports SYSENTER only in Legacy Mode³, not in any of the Long Mode submodes.

• AMD supports SYSCALL in all modes⁴.

So as the OSDEV page on the topic says:

• In 64-bit Long Mode - only SYSCALL works on both ISAs. (SYSENTER doesn't work on AMD.)
• In Legacy Mode - only SYSENTER works on both ISAs. (SYSCALL doesn't work on Intel.)
• There's no single instruction that works on both Intel and AMD in Compatibility Mode (SYSENTER doesn't work on AMD and SYSCALL doesn't work on Intel), but there's no need for one. A 32-bit kernel will stay in Legacy Mode after boot.

---

¹ Intel® 64 and IA-32 architectures software developer's manual volume 2B: Instruction set reference, M-U, p. 4-668

² Intel® 64 and IA-32 architectures software developer's manual volume 2B: Instruction set reference, M-U, p. 4-666

³ AMD64 Architecture Programmer’s Manual Volume 3: General-Purpose and System Instructions, p. 423

⁴ AMD64 Architecture Programmer’s Manual Volume 3: General-Purpose and System Instructions, p. 419

Answer 2

64-bit user-space always uses syscall, across all x86-64 OSes.

32-bit user-space under a 32-bit Windows kernel uses sysenter if available.

WoW64 (32-bit user-space aka Windows, on a 64-bit kernel aka Windows64) uses a call far into a 64-bit ntdll which uses 64-bit syscall.

Some other OSes, such as Linux, do enter a 64-bit kernel directly from 32-bit user-space, with sysenter or syscall depending on which the CPU supports (via the VDSO which the kernel maps into the address-space of user-space processes). See Calling system API from 32-bit processes under Linux 64-bit for more details on syscall vs. sysenter and which modes they're available in on different CPUs, and the fact that legacy-mode (32-bit kernel) syscall is so badly designed (from a Linux kernel perspective at least) that Linux doesn't use it even if that means a fallback to int 0x80.

---

WoW64's call far / retf nearly doubles the cost of getting into the kernel and back for 32-bit user-space (compared to sysenter), but isn't a big fraction of the total time for most system calls. This has always seemed like an inefficient design to me, so I wonder if they chose it only because there's no single instruction that works from 32-bit compat mode under a 64-bit kernel on both Intel and AMD x86-64 CPUs. Or it there's useful stuff that can be done in user-space to avoid calling into the kernel at all, but only with 64-bit code?

A faster system-call instruction is something that OS devs do care about, e.g. Raymond Chen's blog about how on 386, the illegal instruction trap was the fastest way into the kernel so Windows used that.

A call far [mem] / retf pair on i7-6700k Skylake takes 220 core clock cycles when measured in a simple microbenchmark loop. From 32-bit user-space to 64-bit user-space, or from 32 to 32 costs the same. NASM source with perf results for a static Linux executable I used to test by running perf on the whole program. With basically no startup overhead, and running enough iterations to run for over a second, this gives pretty accurate measurements.

A do-nothing system call with an invalid syscall number (EAX=-1) takes 1209 cycles on the same system, Linux kernel 6.5 with Spectre + Meltdown mitigation, including swapping page tables. So an extra call far + retf is about 18% extra cost for a do-nothing system call in the best case with caches hot.

I also tested call far / 64-bit syscall / retf (1439 cycles) vs. int 0x80 (1772 cycles) vs. sysenter (1305 cycles), average cost in a tight loop. This is on x86-64 Linux 6.5 on my i7-6700k Skylake, with EAX=-1, so it returns -ENOSYS without dispatching to a sys_whatever function, but still does a bunch of stuff inside the kernel. (It's optimized for the case of system calls that don't error, so it doesn't check that until it's ready to dispatch to a handler function.) So WoW64's strategy is better than using int 0x2e, at least on Skylake and probably most CPUs. But it's worse than using sysenter on CPUs that support that from compat mode.

The 134 cycle delta between the WoW64 strategy and sysenter isn't as big as call far / retf alone, so maybe 32-bit sysenter is slower than 64-bit syscall, or the Linux kernel internals are different for compat-mode system calls vs. native 64-bit syscalls.

---

call far / retf was somewhat less slow on older CPUs (when 32-bit code was even more common on Windows), for example Agner Fog measured call far [mem] at 79 cycles on Nehalem (microcoded as 47 uops), plus retf taking 120 cycles, for a total of 199 cycles for a pair. Or 153 for call far+retf on Core 2. But only 33 cycles in AMD K8 and K10. Agner didn't measure call far or retf for CPUs newer than that.

32-bit code is still not rare on Windows, where some projects are built around binary-only DLLs. The software ecosystem around other OSes is less binary-centric, and some current Linux distros are even considering disabling running 32-bit executables in the kernel, at least by default.

Answer 3

syscall (created by AMD) and sysenter (created by Intel) are competing implementations of fast switch-to-ring0 instructions. For compatibility, x64 systems support both types. That wasn't deemed to be necessary for 32-bit systems.