Visual studio memory allocation reverse engineering

Question

I compile this code with Visual studio 2010 compiler:

#include "stdafx.h"
#include <iostream>
int main() {
   int *p;
   p = new int(255);
   delete []p;
}

The disassembly of it, is different from Dev C++. It seems it first checks if there is enough memory and then start the allocation. am I right?

This is the disassembly :

In the Orange node:

Why esi and edi pushed to the stack? I've seen mov eax,0CCCCCCCCh before in books, What does this instruction do? What does the highlighted part of the orange node do? Is it a check to see if there is enough available memory?

In the blue node:

FFh is equal to 255, Can you explain how the memory is getting allocated?

Answer 1

You may want to read up on assembly before attempting to reverse engineer.

1. esi and edi are pushed on the stack because the compiler thought this routine modifies them. (It is wrong because only edi is used. Still, better safe than sorry.)

2. mov eax,0cccccccch moves the value 0CCCCCCCCh into register eax. Which is actually kind of self-explanatory. That instruction in itself does nothing particularly useful, and you should be careful to ask such questions. It is clear from the next lines that the value gets stored into the Local Variable area, to fill it with a 'known' value, rather than having random values.

   The value 0CCCCCCCCh is used as a sentinel value and so if the context is "it gets stored somewhere", then its purpose is to catch uninitialized pointers.

3. Again, time for an assembly refresher. The first highlighted line

   add esp, 4
   

   is not part of the following instructions, it's Stack Cleanup for the previous instruction: the call.

   The lines mov [ebp+var_E0], eax and cmp [ebp+var_E0], 0 have nothing to do at all with any kind of "allocation" or "memory"! All it does is save eax – the return value of the previous call – into a local variable, and then test if the value is 0. That is boilerplate generated code for

   var_E0 = new (uint);
   if (var_E0 == 0)
      ...
   

   which is the only 'check' there is, and only after attempting to allocate, not before.

4. 'Blue node': the code in assembler does what the C++ code is supposed to do. It allocated space for a single integer before (the push 4 in the call to new) and in the blue node, it stores the value 255 into the newly allocated memory. If you expected it to allocate 255 bytes: well no. It does what the C++ code is supposed to do, which is explained in What does "new int(100)" do? and the question of which it is marked a duplicate.