What is *(_BYTE *), __usercall, eax and edi [closed]

Question

I am trying to understand some algorithm of how the thing works. Here is the function I got from decompiling my executable with Hey Rays.

I want to run this function to see the result and then understand how it works.

So the first thing I want to know what all the signature of this function means?

__usercall sub_401960<eax>(int a1<edi>)
__usercall
eax and diamond <>
edi and diamond <>

Then I want to know what these things mean:

*(_BYTE *)

and

__security_cookie

and what they're doing in the following code:

#include <stdio.h>
#include <stdlib.h>


int __usercall sub_401960<eax>(int a1<edi>)
{
  int result; // eax@1
  signed int v2; // esi@1
  char v3; // cl@2
  char v4; // cl@3
  char v5; // cl@8
  char v6; // cl@9
  char v7; // cl@14
  char v8; // cl@15
  char v9; // cl@20
  char v10; // cl@21

  result = 0;
  v2 = 0;
  do
  {
    v3 = *(_BYTE *)(v2 + a1);
    if ( (unsigned __int8)(v3 - 48) > 9u )
    {
      if ( (unsigned __int8)(v3 - 97) > 5u )
      {
        if ( (unsigned __int8)(v3 - 65) > 5u )
          return result;
        v4 = v3 - 55;
      }
      else
      {
        v4 = v3 - 87;
      }
    }
...
  return result;
}

Sorry if I've asked such newbie question

What I want is to run the thing!

Answer 1

First, OP mentioned several times he'd like to execute the function. Which should be pretty easy using a debugger (IDA has decent debugging capabilities, or you could give ollydbg a try)

I'll answer the questions in the order in which they appear:

what is __usercall and diamond-wrapped registers in a function's definition?

__usercall is a "virtual calling convention" in IDA. This is not a real world calling convention, but instead it let an IDA user to specifically describe how parameters are passed to the function. This is particularly useful when parameters are passed in some unconventional way, and you can't use either __stdcall, __fastcall, etc.

The diamond wrapped register names in the prototype are used in __usercall to assign a specific register to that variable (or return address) in the function's body.

The prototype you've got (__usercall sub_401960<eax>(int a1<edi>)) means the return value is placed in eax and a1 is passed to the function in edi.

What does *(_BYTE *) mean?

This actually has two distinctive parts, evaluated one after the other.

First, (_BYTE *) casts a value or a register to be a byte pointer. This is similar to assigning the value to a C variable which is defined as byte *.

Second, * dereferences the address and retrieves the value in that address, value is assumed to be of the type of the pointer, in our case _BYTE.

This is plain C syntax, read more about it online. Good C understanding is very beneficial to reverse engineers.

What is __security_cookie?

__security_cookie is a variable defined in Microsoft's Visual Studio's compiler when stack canaries are used as stack protections. These are variables that are set to a specific (non deterministic) value at the start of the function, and are validated for the same value when the function ends.

This is done to make sure no one managed to overflow a buffer defined in the function's stack, and overwrote return addresses. Because such an overflow must also override the __security_cookie which has an unknown value.

Answer 2

*(BYTE *) means take a byte from the pointer provided in address

if address is 12345678 and it holds hello

then x = *(byte *) (12345678 + counter) if counter is 0 x will be 'h' and so on

the truncated part checks for [0..9a..eA..E] and returns either 0 (result or 10,9,8,7,6 to v4
don't see any reference to __security_cookie in Your query

but if you were asking in general what it meant then you should read about stack canaries , stack smashing protection , gscookie etc

it is a part of implementation to protect against stack / buffer overflow exploits / vulnerabilities