1

I'm embarking on my first reverse-engineering adventure. I think I've picked a rather dense piece of steak to start out with.

I have three C programs which communicate together using an unknown IPC protocol over TCP sockets.

  • A Linux armel binary that might be described as the "target".

  • A Linux x86 binary that provides a "shell" or REPL that translates text commands on stdin to IPC I/O against the target. This is a tiny 50KB binary (probably because it was compiled with -O3 and then stripped ;D).

  • A second Linux armel binary that presents a frontend/UI to the target and incorporates additional IPC-related functionality not found in the REPL.

My first goal is to understand the fundamentals of the IPC wire format by disassembling the shell/REPL control program. The smallness of the control program is encouraging.

With this done, I want to dive into the additional IPC-related functionality only available in the frontend program, which is somewhat bigger (and has noise in the form of UI code).

I have a very hopeful starting environment: I've gotten the two armel programs running in QEMU (the target, easily enough; the control program, after much effort). The x86 binary trivially runs on my host computer.

I've also put together a shaky but functional hamster dance of scripts that route the traffic between the two armel programs (incidentally running in separate QEMU VMs, for operational simplicity!) in such a way that I can intercept the traffic at any time.

And so it is that I... take one look at the protocol and have no idea how to proceed:

21:37:21.839965  UI->target   08 00 00 00  01 00 00 01  00 00 00 00  03 00 00 00   .... .... .... ....
                 UI->target   01 00 00 00                                          .... 

21:37:21.892845  target->UI   04 00 00 00  01 00 00 01  00 00 00 00  3a 03 00 00   .... .... .... :...

21:37:21.930144  UI->target   08 00 00 00  06 00 00 01  00 00 00 00  6c 70 70 61   .... .... .... lppa
                 UI->target   00 00 00 00                                          ....

21:37:21.991124  target->UI   d0 02 00 00  06 00 00 01  00 00 00 00                .... .... ....

I incidentally know what "lppa" means - but this small fragment was taken from a 4,000-line protocol dump that clearly shows that the other bytes change in important ways, but I don't have the experience to discern patterns or significance from the changes.

The only thing I immediately twigged about was the suspicion that I was looking at C structs dumped straight onto a network. For what this program was and did, that makes a lot of sense (it's for all intents and purposes an internal IPC protocol that doesn't, well, need to be particularly resilient).

So, I fired up Ghidra, which I learned about a little while back. Finally, IDA Pro has some competition :)

I spent some time ambling aimlessly around inside the decompiler, and fairly quickly found the following interesting function that does both recv() and send().

But the way this function works is both really interesting and really confusing.

I've highlighted some lines below that refer to buffer_length which are really messing with my head.

When I first saw the code around buffer_length I happened to be looking at the second send() call, so it felt natural to rename that variable to describe "length".

But then I looked at the first send() and found that buf was being pointed to... yeah no that doesn't work.

So I now wonder if this might actually a union. YAY.

(See below for the struct definition)

uint maybe_perform_ipc(astruct *something_struct) {

    char *buf;
    uint return_code;
    ssize_t bytecount;
    size_t message_length;

    bytecount = 0;
    return_code = 0x30040000;
    something_struct->only_used_once_and_set_to_256 = 0x100;
    message_length = 12;
    buf = (char *)&something_struct->buffer_length;

    while (
        message_length > 0 &&
        (bytecount = send(something_struct->socket, buf, message_length, MSG_NOSIGNAL)) > 0
    ) {
        message_length -= bytecount;
        buf += bytecount;
    }

    if (bytecount > -1) {

        if (something_struct->buffer_length != 0) {
            message_length = something_struct->buffer_length;
            buf = (char *)something_struct->buffer;
            while (
                message_length > 0 &&
                (bytecount = send(something_struct->socket, buf, message_length, MSG_NOSIGNAL)) > 0
            ) {
                message_length -= bytecount;
                buf += bytecount;
            }
            if (bytecount buf = (char *)&something_struct->buffer_length;

        while (
            message_length > 0 &&
            (bytecount = recv(something_struct->socket, buf, message_length, 0)) > 0
        ) {
            message_length -= bytecount;
            buf = buf + bytecount;
        }

        if (bytecount > -1 && message_length maybe_status_code == 0) {

                my_free_then_realloc((astruct_4 *)something_struct, something_struct->buffer_length);

                if (something_struct->buffer_length != 0) {

                    message_length = something_struct->buffer_length;
                    buf = (char *)something_struct->buffer;

                    while (
                        message_length > 0 &&
                        (bytecount = recv(something_struct->socket, buf, message_length, 0)) > 0
                    ) {
                        message_length -= bytecount;
                        buf = buf + bytecount;
                    }

                    if (bytecount maybe_status_code | 0x30060000;

            }

        }

    }
    return return_code;
}
offset  length  type           name
0x0     0x4      int           maybe_signature 
0x4     0x1      undefined  
0x5     0x1      undefined  
0x6     0x1      undefined  
0x7     0x1      undefined  
0x8     0x1      undefined  
0x9     0x1      undefined  
0xa     0x1      undefined  
0xb     0x1      undefined  
0xc     0x4      int           socket 
0x10    0x4      uint          buffer_length 
0x14    0x2      undefined2    maybe_function_code 
0x16    0x2      undefined2    only_used_once_and_set_to_256 
0x18    0x2      ushort        maybe_status_code 
0x1a    0x1      undefined  
0x1b    0x1      undefined  
0x1c    0x4      uint  
0x20    0x4      void *        buffer 
0x24    0x4      undefined4    maybe_pid 

I couldn't figure out how to make Ghidra export the above as a struct so I've just dumped it here as Ghidra presented it, since it's quite readable as it is (and I have no idea how I'd rewrite this as a C struct myself, actually :/).

Looking at buffer_length, if I understand how unions work properly, a function code and a status code (guess-identified from elsewhere) would be kind of exactly the kind of thing you'd want to dump out of a C struct into a network packet.

ALSO, if you go back and look at the code where it sets message_length to 12, the range from buffer_length extends down to the two undefineds currently underneath maybe_status_code.

I unfortunately don't know how to tell Ghidra "go and find me where these two definitions are used!" from the struct editor, and I don't know how to tell Ghidra "search in decompiled code" for field_0x1a and field_0x1b, so these two remain mysteries for now. (I also tried to turn the range into a sub-struct, but then it disappeared in the struct editor.)

Anyway, my point, and my question is...

What direction should I head in so I can make progress with this dissection and get enough context to make sense of what I'm seeing?

Static analysis is proving really cool and interesting, but only because I can decompile to C. Ghidra doesn't provide debugger integration :'(, so I can't fire the shell program up and watch it tick away unless I want to stare at assembly in GDB.

Which I'm not at all unwilling to do, I just find asm really disorientating. I must admit I've been trying to pick it up for about... wow, I think it's been 15 years now, heh. (All the tutorials I've yet found are either really regimented (to the point I can't piece the details together), assume I'm a vegetable (...8086? uhh....), or try and teach me to juggle by handling me 6 balls to start with.)

It's very late so I'll leave this question here for now, and possibly add to it / clarify details a little later on.

One key point: the program I'm reverse engineering is proprietary but its status is "quietly floating around on the internet if you know what rock to look under", so I could share more details privately (incidentally my email is in my bio).

  • I think the trick about those undefined is alignment. The variable at 0x1C is an integer, so it is uses 4-byte alignment; since maybe_status_code is a short (only 2 bytes), 2 bytes of padding have to be added to align the next variable. So maybe fields 0x1A and 0x1B don't do anything at all. – eric.m Jul 4 at 7:36
  • About the mesage length thing, I think the code is sending the data from buffer_length to maybe_status_code as some kind of signature; after all, the receiving code must know how many bytes the buffer has, the function code, etc – eric.m Jul 4 at 7:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.