What is a good resource to learn about how the call stack works while programming in C?

Question

Sorry if this is a duplicate and please point me in the right direction if so. I have a strong foundation in C programming and I know how memory is allocated on the stack for variables/arrays/pointers (of all datatypes) etc. I also understand how to use malloc and free to put data on the heap. What I do NOT understand is how all the registers and the call stack fit together with this picture and what each registers is supposed to point to. I am trying to learn this before I start messing with GDB so I actually know what I am looking at. Any good references out there?

Ultimately this is so I can do a project for my masters where I need to run shellcode by exploiting a buffer overflow vulnerability in a given program. I know how to do a basic buffer overflow by feeding the program more data than allocated but I don't know what/how much data I need to feed the program so that it gets put at the right place in memory in order to execute what is required.

Answer 1

What I do NOT understand is how all the registers and the call stack fit together with this picture and what each registers is supposed to point to.

It's great that you are competent in C, but there is no way around reading disassembled object code and writing assembly code (practical experience).

The compiler toolchain is responsible for generating object code that targets a particular CPU. This means that it is the compiler that is responsible for managing all of the registers based on details such as process layout in memory (which is OS-specific), calling convention, optimization level, application binary interface (also OS-specific), etc.

You did not give any information about the environment you are working in aside from mentioning GDB, so I am guessing you are going to be doing Linux x86 binary exploitation.

• A good theoretical foundation for understanding the stack is provided by Chapter 8 "Subroutines and Control Abstraction" in Programming Language Pragmatics by Michael L. Scott.

• A good practical introduction to Linux x86 stack structure and management is provided in section 3.7 "Procedures" in Computer Systems: A Programmer's Perspective by Bryant and O'Hallaron.

Lastly, there is the following Q&A regarding the stack over at the CS.SE site:

How are variables stored in and retrieved from the program stack?