Why does program fail to run under several debuggers?

Question

To lay out the obvious for a start, I am a beginner in the Reverse Engineering area when it comes to Computer Science. I have come across a program which I wanted to reverse engineer for myself to see how it works and what I can tamper with and how (meaning I have not got the source code). I will not be giving the name of the software for obvious reasons, so I will refer to the software as 'X' (how generic).

So the case is, when I start IDA Pro and put 'X' in it, it manages to load up successfully the subroutines, however, many library functions are apparently left unnamed, such as "unnamed_libname_[number]" and I find difficulties searching through them and analysing them. In addition, Strings are barely existing - only function names of the Windows API, such as IsDebuggerPresent and ReleaseMutex, where they appear, but the problem that I am struggling with comes even earlier.

When I run 'X' after I've loaded it (it is a 32-bit .exe should that matter), the program very quickly leads to an error which is: "[address]: The instruction at [address] referenced memory at 0x0. The memory could not be written -> [plenty of zeros] (exc.code c0000005, tid [id]" and the instruction it fails at is apparently

security.dll : [address] mov     byte ptr [eax], 0

I tried backtracing, even though I find it difficult, especially as I do not know what to look for or what to expect, so then I tried a couple of other tools to help understanding this better and realizing what the problem is.

Next debugger I tried was OllyDbg, where after I load and run the .exe, OllyDbg simply closes and then nothing happens. No .log file, no error message, nothing. When I try attaching it, my breakpoints don't work and one function of OllyDbg is 'Execute until return', which I use when I want to press a button. Nothing happens and Olly does not halt the process, even though it MUST have received probably a dozen returns.

Obviously something weird is going on.

Then I open up Dependency Walker as it might provide some information on the .dlls used, their functions, etc, hopefully being able to find out something about 'security.dll', which in fact has almost no results on google concerning this .dll, located in SysWOW64. Upon loading the .exe in DW, it takes a bit of time and after it's ready loading, it throws hundreds of errors such as "Error opening file. The system could not find the file specified." which according to some is not a problem - these are delay-load dependencies which get loaded during runtime and DW would not initially know where they are located... OK! People suggested profiling it, so I try doing that. As I now profile the application under Dependency Walker (using the default settings) and run it, an "Application Error" window is thrown, saying "The application was unable to start correctly ([address]). Click OK to close the application." and in the log file it gives the following errors:

Started "[program name].EXE" (process 0x2AF4) at address 0x00400000.  Cannot hook module.
Loaded "NTDLL.DLL" at address 0x77850000.  Cannot hook module.
Loaded "KERNEL32.DLL" at address 0x759D0000.  Cannot hook module.
Loaded "KERNELBASE.DLL" at address 0x747E0000.  Cannot hook module.

Followed by:

DllMain(0x747E0000, DLL_PROCESS_ATTACH, 0x00000000) in "KERNELBASE.DLL" called.
DllMain(0x747E0000, DLL_PROCESS_ATTACH, 0x00000000) in "KERNELBASE.DLL" returned 1 (0x1).
DllMain(0x759D0000, DLL_PROCESS_ATTACH, 0x00000000) in "KERNEL32.DLL" called.
DllMain(0x759D0000, DLL_PROCESS_ATTACH, 0x00000000) in "KERNEL32.DLL" returned 1 (0x1).
Injected "DEPENDS.DLL" at address 0x08370000.
DllMain(0x08370000, DLL_PROCESS_ATTACH, 0x00000000) in "DEPENDS.DLL" called.
DllMain(0x08370000, DLL_PROCESS_ATTACH, 0x00000000) in "DEPENDS.DLL" returned 1 (0x1).
Exited "[program name].EXE" (process 0x2AF4) with code -1073741502 (0xC0000142).

And at this point I am completely clueless as to what I can/should do. Most probably all of this is a symptom of something, which means something deeper in the program is causing all of this, but I have no idea what it might be, where it might be and how one can analyze and fix it.

My belief is that it may have something to do with an anti-debugging technique, as the program runs fine without trying all of these fancy reverse things, but as I said before, I'm clueless.

If anyone happens to have any idea what might be causing this behavior in several debuggers, the symptoms that arise and ways I can possibly tackle this situation, it would be of great help!

In addition, what is a reason why strings may not appear in the disassembly windows, such as "Options" or a message being given to the user? No other files exist in the directory that may possibly have such data.

I have also had the thought that it may have something to do with obfuscation/packing, although PEiD gives no results when I scan it plus it is a bit illogical to be impossible to run normally under a debugger such as IDA and OllyDbg.

All answers appreciated heartily!

Answer 1

Regarding security.dll, it's the standard Windows DLL for SSPI functions. Regarding program X, given that it doesn't work when run under a debugger and doesn't contain human-readable strings, it strongly suggests that it's packed and that the unpacking stub and/or the original program contains anti-debugging code. This is fairly common.

It's possible that the exception thrown by security.dll was a first-chance exception unrelated to anti-debugging and that there was code in security.dll to handle it. But if it was related to your program's anti-debugging code, then perhaps it was made to look like an exception in security.dll to act as a red-herring and slow down analysis.

Most likely it is packed and using several tools, specifically the .exe file is packed with Armadillo v.8 or higher. Armadillo has a debug-blocker feature with a built-in security.dll file, not affiliated with the Windows security.dll file.

Armadillo use own security.dll (loaded from memory) and some anti-debugging. Some version create two processes - and second process debug first - on exceptions - unpacking chunk of code in first process.

^{Answer based on the comments above.}

Answer 2

Welcome to the world of reverse engineering! What you are facing is differents sorts of anti-debugging techniques. You mentioned "IsDebuggerPresent and ReleaseMutex". Those are records in the windows registry and they get enabled when you run a program in debug mode in Windows. They are usually linked with some routine that will cause debugging difficulties, such as the instruction failure you faced (0x0) and many other things that could also result in not letting the code execute at all. Those things are usually caused by some programs called packers, which work before the main program does. Manual unpacking need lots of research and experience. Since you are a beginner I would suggest you start working with simple cmd programs in C/C++ or any other language can build you executables for you to debug. Get familiar with the assembly, and the way your OS communicates with the memory and the processor. Start working with packers and packed executables after getting some experience and confidence. Last, but not least, I will give you the link of the best free book for reversing out there. Read some stuff or even keep it as a reference.

Yours sincerely, a fellow Computer Science student.