How calls work in x86

Question

I am studying some X86 code and I often see calls:

call sym.imp.printf
call sym.imp.scanf
call sym.imp.strcmp
call sym.imp.__stack_chk_fail

Those examples are the most common calls. But how do they actually work? I mean, I know they are system calls, also printf along with scanf, strcmp are C functions. But my question is where do they get the parameters from?

sym.imp.strcmp: where is it getting strings from to compare?

Where is the value of scanf saved?

And also, what does the call sym.imp.__stack_chk_fail does?

Answer 1

System calls vs. function calls

I mean, I know they are system calls, also printf along with scanf, strcmp are C functions.

Many C library functions are wrappers around system calls. printf and scanf are are examples of this. However, it should not be assumed that all C library functions execute system calls, as none of the string.h library functions, including strcmp, execute any system calls.

A system call is a controlled entry point into the kernel, allowing a process to request that the kernel perform some action on the process’s behalf. The kernel makes a range of services accessible to programs via the system call application programming interface (API).¹

The mechanism by which system calls are made is quite different than by that which function calls are made:

The [C library] wrapper function executes a trap machine instruction (int 0x80), which causes the processor to switch from user mode to kernel mode and execute code pointed to by location 0x80 (128 decimal) of the system’s trap vector.

More recent x86-32 architectures implement the sysenter instruction, which provides a faster method of entering kernel mode than the conventional int 0x80 trap instruction. The use of sysenter is supported in the 2.6 kernel and from glibc 2.3.2 onward.¹

Here is a visual depiction of the C library function execve being executed, in which execve makes a system call:

x86 calling conventions

When a function is called, flow of control branches to a different location in memory via the call instruction:

Saves procedure linking information on the stack and branches to the procedure (called procedure) specified with the destination (target) operand. The target operand specifies the address of the first instruction in the called procedure. This operand can be an immediate value, a general purpose register, or a memory location.²

Here is some simple example code:

0804841d <main>:
 804841d:       55                      push   %ebp
 804841e:       89 e5                   mov    %esp,%ebp
 8048420:       83 e4 f0                and    $0xfffffff0,%esp
 8048423:       83 ec 20                sub    $0x20,%esp
 8048426:       c7 44 24 18 f0 84 04    movl   $0x80484f0,0x18(%esp)
 804842d:       08 
 804842e:       c7 44 24 1c 04 00 00    movl   $0x4,0x1c(%esp)
 8048435:       00 
 8048436:       8b 44 24 18             mov    0x18(%esp),%eax
 804843a:       89 44 24 08             mov    %eax,0x8(%esp)         <--- argument 3
 804843e:       8b 44 24 1c             mov    0x1c(%esp),%eax
 8048442:       89 44 24 04             mov    %eax,0x4(%esp)         <--- argument 2
 8048446:       c7 04 24 0a 85 04 08    movl   $0x804850a,(%esp)      <--- argument 1
 804844d:       e8 9e fe ff ff          call   80482f0 <printf@plt>   <--- function call
 8048452:       b8 00 00 00 00          mov    $0x0,%eax
 8048457:       c9                      leave  
 8048458:       c3                      ret

Here, the memory address that execution branches to when printf is called via call is 0x80482f0.

But my question is where do they get the parameters from?

Arguments are pushed onto the stack in reverse order of their corresponding parameters in the function definition prior to the function call. The return value is saved in %eax. This is in accordance with x86 calling convention, referred to as cdecl:

Caller Rules

To make a subrouting call, the caller should:

1. Before calling a subroutine, the caller should save the contents of certain registers that are designated caller-saved. The caller-saved registers are EAX, ECX, EDX. Since the called subroutine is allowed to modify these registers, if the caller relies on their values after the subroutine returns, the caller must push the values in these registers onto the stack (so they can be restore after the subroutine returns.

2. To pass arguments to the subroutine, push them onto the stack before the call. The arguments should be pushed in inverted order (i.e. last argument first). Since the stack grows down, the first arguments will be stored at the lowest address (this inversion of arguments was historically used to allow functions to be passed a variable number of parameters).

3. To call the subroutine, use the call instruction. This instruction places the return address on top of the arguments on the stack, and branches to the subroutine code. This invokes the subroutine, which should follow the callee rules below.

After the subroutine returns (immediately following the call instruction), the caller can expect to find the return value of the subroutine in the register EAX. To restore the machine state, the caller should:

1. Remove the arguments from stack. This restores the stack to its state before the call was performed.
2. Restore the contents of caller-saved registers (EAX, ECX, EDX) by popping them off of the stack. The caller can assume that no other registers were modified by the subroutine. ³

For a more in-depth discussion of x86 calling conventions, refer to the x86 ABI documentation found in the System V Application Binary Interface Intel386 Architecture Processor Supplment, Fourth Edition.

__stack_chk_fail and stack guards

And also, what does the call sym.imp.__stack_chk_fail does?

__stack_chk_fail is called when the stack canary has been overwritten due to a buffer overflow:

The basic idea behind stack protection is to push a "canary" (a randomly chosen integer) on the stack just after the function return pointer has been pushed. The canary value is then checked before the function returns; if it has changed, the program will abort. Generally, stack buffer overflow (aka "stack smashing") attacks will have to change the value of the canary as they write beyond the end of the buffer before they can get to the return pointer. Since the value of the canary is unknown to the attacker, it cannot be replaced by the attack. Thus, the stack protection allows the program to abort when that happens rather than return to wherever the attacker wanted it to go.⁴

Here is some example annotated code:

000000000040055d <test>:
  40055d:   55                      push   %rbp
  40055e:   48 89 e5                mov    %rsp,%rbp
  400561:   48 83 ec 20             sub    $0x20,%rsp
  400565:   89 7d ec                mov    %edi,-0x14(%rbp)
  400568:   64 48 8b 04 25 28 00    mov    %fs:0x28,%rax     <- get guard variable value
  40056f:   00 00 
  400571:   48 89 45 f8             mov    %rax,-0x8(%rbp)   <- save guard variable on stack
  400575:   31 c0                   xor    %eax,%eax
  400577:   8b 45 ec                mov    -0x14(%rbp),%eax
  40057a:   48 8b 55 f8             mov    -0x8(%rbp),%rdx   <- move it to register
  40057e:   64 48 33 14 25 28 00    xor    %fs:0x28,%rdx     <- check it against original
  400585:   00 00 
  400587:   74 05                   je     40058e <test+0x31>
  400589:   e8 b2 fe ff ff          callq  400440 <__stack_chk_fail@plt> 
  40058e:   c9                      leaveq 
  40058f:   c3                      retq   

---

_{1. The Linux Programming Interface, Chapter 3 "System Programming Concepts"}

_{2. x86 Instruction Set Reference - CALL - c9x.me}

_{3. x86 Assembly Guide - University of Virginia Computer Science}

_{4. "Strong" stack protection for GCC - LWN.net}