Reversing a Mac OS X binary that appears to be non encrypted, backtrace just shows mach_msg_trap ()

Question

Currently I have a binary that I am investigating. The application is GUI / event driven, so that makes it difficult to set a break point. I would like to set a break point on a certain button click, so I thought I would click the button, and then run a backtrace in GDB to see what functions were called when I clicked the button, but the output of the bt is just showing mach_msg_trap(), and a few other "functions" I suppose. Does anyone know why I'm in the mach_msg_trap() I am assuming it's some security feature implemented by Apple to prevent people from reversing their software, I just thought I would ask, as my googlefu didn't really return any tangible results.

Answer 1

Well, my Mac fu isn't too strong, but I have terminal access to a Mac and will give it a wild shot, hoping to provide you with the insight required to proceed.

info sharedlibrary on the GDB prompt will tell you details about the ranges of the shared libraries inside of which those functions reside. But we know that already from the function names.

NSApplication comes from: Cocoa.

The NSApplication object maintains a list of all the NSWindow objects the application uses, so it can retrieve any of the application’s NSView objects.

So my suggestion would be to start by starting the application

gdb --args /path/to/binary optional arguments for program

and then set a breakpoint:

b NSApplicationMain

(can be done inside .gdbinit obviously)

Then let the application continue. Break into it using Ctrl+C. Most likely the backtrace will be similar to what you saw before, but probably not up to mach_msg_trap().

Then you go to one frame that uses a function which has a reference to the NSApplication instance. You can find the frame numbers with bt, obviously. So say:

f 9

there you should be able to use GDB's call. Now, my Objective-C/C++ fu is even weaker than my Mac fu, but you can use

set language

to list the available languages (which influences the syntax for call). So you should probably switch to one of the Objective-C/C++ alternatives as appropriate. On my Mac I have the following language options:

(gdb) set language
The currently understood settings are:

local or auto    Automatic setting based on source file
ada              Use the Ada language
c                Use the C language
c++              Use the C++ language
asm              Use the Asm language
minimal          Use the Minimal language
fortran          Use the Fortran language
objective-c      Use the Objective-c language
objective-c++    Use the Objective-c++ language
java             Use the Java language
modula-2         Use the Modula-2 language
pascal           Use the Pascal language
scheme           Use the Scheme language

call lets you call functions inside the context of the running program (and GDB has command line completion for symbols it knows of). So calling mainWindow of the NSApplication instance would appear as a good idea and then drilling down from there.

From there you should be able to figure out where your button comes into play and set the breakpoint, no?

Basically if you know a certain parameter of a certain call in the framework references your button (we'll assume it has a pointer somehow) you can use the

break location if condition

syntax to make sure the breakpoint will only trigger for the button press. An alternative is the ignore command which will ignore X hits of a breakpoint and only trigger after that.

Answer 2

Do you have md5 for the application?

It seems like a GUI/Cocoa application. The code seems to be stuck in msgloop. This loop generally occurs in applications, which require user interaction or when application is interacting with messages.

The image shows the breakpoint has been hit inside __CFRunLoopRun. It means app was running and processing system messages until it met a breakpoint (which is Command+C, in this case) it need some kind of user interaction or is waiting for some message, may be a click or key press or message from some system process.

If you have the sample md5 or SHA, I can have a look at it