Reversing a Windows service

Question

I am trying to reverse engineer a malware that open a windows service dynamically in OllyDbg. When the malware calls StartServiceCtrDispatcherW, I receive an error:

How I can continue to analyze this malware dynamically?

Note: I already used Image File Execution Options

Answer 1

If you want to debug the service initialization and it happens automatically (not triggered by some action you perform), you probably can't do it on any Windows newer than XP with ollydbg. You'll have to use WinDbg.

You need to set IFEO Debugger for your process name to run CDB as server (e.g. cdb.exe -server tcp:port=12345 -noio) and the run WinDbg as a client and connect to your server (windbg.exe -remote tcp:server=localhost,port=12345).

You'll probably want to change the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\ServicesPipeTimeout Registry value to be a bit longer. This is the time the SCM waits for a service it runs to talk to it.

If you don't have to debug the initialization you can simply attach to the service after it starts, and then you can use ollydbg.

All of this and pretty much everything you need to know is documented under the MSDN page titled Debugging a Service Application.

---

Edit: If patching the binary is possible you can add an infinite loop in the entry point like gandolf suggested and then attach a debugger after you log in.

Or, if the binary doesn't do any SEH tricks that interfere with it: Add an exception to the entry point (0xCC - int 3 is the obvious choice), set AeDebug to a long running process (such as notepad.exe) and then attach a debugger. This is what Inside Windows Debugging proposes (p. 139).

Or, what's even easier and makes even more sense: Add the same INT3, and set AeDebug to the same cdb.exe command-line as you would put in IFEO and connect WinDbg to it after you log in.

Answer 2

Well, as far as I know you can't debug service's handling code inside the debugger(proof).

If you really need it, I think you should stick with kernel debug of vm with malware running, but maybe someone will offer better solution - this one is the first that came to mind.

UPD

There is a link in comments that covers needed basics of kernel debugging. However, after you finish with setup of all the related things, you might face with the problem of breaking on exact place where service gets started.

I think following command will help you to go further unless you did it yourself already:

bp ADVAPI32!StartServiceCtrlDispatcherW; dd esp or use this cheat sheet for more inspiration with Windbg.