9

In a microkernel, much of the interesting functionality happens not with traditional function calls, but instead through message passing between separate entities.

Is there a structure that OS architectures like this generally use to implement message passing? And is there a methodical way to label this while disassembling, to make it as easy to follow paths of messages as it is to follow the call stack?

9

I don't think I disassembled any microkernels, but "message passing" is common in at least two categories of programs: Win32 GUI (both raw Win32 and MFC-based), and Objective-C executables.

In both cases you have some kind of a central dispatcher routine (or routines) that accept messages and forward them to some recipients, which may be inside or outside the current program.

The recipients can be registered dynamically (RegisterClass in Win32) or may be specified in some static fashion (Objective-C class metadata or MFC's message handler tables).

As for dispatchers, let's consider Win32's SendMessage. It has arguments hWnd and Msg (and extra parameters). The first specifies the recipient. You may be able to trace where it came from and then just look up the class registration corresponding to the window and check whether its window procedure handles this specific message. I guess you could mark the call with a comment "goes to window procedure 0x35345800" or similar to keep track of it. With MFC you'll need to find the class' message table and look up the corresponding handler.

With Objective-C, objc_msgSend accepts the receiving object and the selector to perform. If you can track back the the object, you can check if it has the selector with that name. Or, alternatively, check all selectors with this name in the program. Again, once you found it, make a comment.

So, a similar approach can probably be extended to all other message-passing systems - find recipients, then at the place of the dispatcher call check which ones can potentially handle it, and check the handlers. Sometimes you don't even need to actually do the first part - if the message ID/name is unique enough, you may be able to find the handler just by searching for it.

A somewhat related problem is working with C++ and virtual functions but it has been covered in another question.


I've just remembered one more kind of programs that don't use the call stack. It's those that use Continuation-passing Style, usually written in some kind of a functional language. Greg Sinclair wrote a very nice and entertaining paper on the horrors of disassembling CHICKEN - an implementation of the language Scheme. His site is down but luckily Archive.org kept a copy. One quote from it:

For a reverse engineer, Continuation Passing Style means the end of civilianization as we know it.

2
  • All kinds of IPC on various OSs also use a similar method. E.g. RPC and LPC on Windows. Heck, couldn't IRPs in Windows kernel mode be categorized here as well? – 0xC0000022L Mar 23 '13 at 0:05
  • Indeed, IRPs is another possibility! – Igor Skochinsky Mar 23 '13 at 3:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.