To lay out the obvious for a start, I am a beginner in the Reverse Engineering area when it comes to Computer Science. I have come across a program which I wanted to reverse engineer for myself to see how it works and what I can tamper with and how (meaning I have not got the source code). I will not be giving the name of the software for obvious reasons, so I will refer to the software as 'X' (how generic).
So the case is, when I start IDA Pro and put 'X' in it, it manages to load up successfully the subroutines, however, many library functions are apparently left unnamed, such as "unnamed_libname_[number]" and I find difficulties searching through them and analysing them. In addition, Strings are barely existing - only function names of the Windows API, such as IsDebuggerPresent and ReleaseMutex, where they appear, but the problem that I am struggling with comes even earlier.
When I run 'X' after I've loaded it (it is a 32-bit .exe should that matter), the program very quickly leads to an error which is: "[address]: The instruction at [address] referenced memory at 0x0. The memory could not be written -> [plenty of zeros] (exc.code c0000005, tid [id]" and the instruction it fails at is apparently
security.dll : [address] mov byte ptr [eax], 0
I tried backtracing, even though I find it difficult, especially as I do not know what to look for or what to expect, so then I tried a couple of other tools to help understanding this better and realizing what the problem is.
Next debugger I tried was OllyDbg, where after I load and run the .exe, OllyDbg simply closes and then nothing happens. No .log file, no error message, nothing. When I try attaching it, my breakpoints don't work and one function of OllyDbg is 'Execute until return', which I use when I want to press a button. Nothing happens and Olly does not halt the process, even though it MUST have received probably a dozen returns.
Obviously something weird is going on.
Then I open up Dependency Walker as it might provide some information on the .dlls used, their functions, etc, hopefully being able to find out something about 'security.dll', which in fact has almost no results on google concerning this .dll, located in SysWOW64. Upon loading the .exe in DW, it takes a bit of time and after it's ready loading, it throws hundreds of errors such as "Error opening file. The system could not find the file specified." which according to some is not a problem - these are delay-load dependencies which get loaded during runtime and DW would not initially know where they are located... OK! People suggested profiling it, so I try doing that. As I now profile the application under Dependency Walker (using the default settings) and run it, an "Application Error" window is thrown, saying "The application was unable to start correctly ([address]). Click OK to close the application." and in the log file it gives the following errors:
Started "[program name].EXE" (process 0x2AF4) at address 0x00400000. Cannot hook module. Loaded "NTDLL.DLL" at address 0x77850000. Cannot hook module. Loaded "KERNEL32.DLL" at address 0x759D0000. Cannot hook module. Loaded "KERNELBASE.DLL" at address 0x747E0000. Cannot hook module.
DllMain(0x747E0000, DLL_PROCESS_ATTACH, 0x00000000) in "KERNELBASE.DLL" called. DllMain(0x747E0000, DLL_PROCESS_ATTACH, 0x00000000) in "KERNELBASE.DLL" returned 1 (0x1). DllMain(0x759D0000, DLL_PROCESS_ATTACH, 0x00000000) in "KERNEL32.DLL" called. DllMain(0x759D0000, DLL_PROCESS_ATTACH, 0x00000000) in "KERNEL32.DLL" returned 1 (0x1). Injected "DEPENDS.DLL" at address 0x08370000. DllMain(0x08370000, DLL_PROCESS_ATTACH, 0x00000000) in "DEPENDS.DLL" called. DllMain(0x08370000, DLL_PROCESS_ATTACH, 0x00000000) in "DEPENDS.DLL" returned 1 (0x1). Exited "[program name].EXE" (process 0x2AF4) with code -1073741502 (0xC0000142).
And at this point I am completely clueless as to what I can/should do. Most probably all of this is a symptom of something, which means something deeper in the program is causing all of this, but I have no idea what it might be, where it might be and how one can analyze and fix it.
My belief is that it may have something to do with an anti-debugging technique, as the program runs fine without trying all of these fancy reverse things, but as I said before, I'm clueless.
If anyone happens to have any idea what might be causing this behavior in several debuggers, the symptoms that arise and ways I can possibly tackle this situation, it would be of great help!
In addition, what is a reason why strings may not appear in the disassembly windows, such as "Options" or a message being given to the user? No other files exist in the directory that may possibly have such data.
I have also had the thought that it may have something to do with obfuscation/packing, although PEiD gives no results when I scan it plus it is a bit illogical to be impossible to run normally under a debugger such as IDA and OllyDbg.
All answers appreciated heartily!