Locating Memory Allocation Functions from Static Analysis

Question

I'm currently working on reversing a firmware binary for an embedded device. This device is using an uncommon architecture: it is a modified Harvard architecture, running on a TMS320C55x chipset with an old Real-time Operating System (RTOS) which does not exists anymore (Byte-BOS) and for which I cannot find any manual (if you have, please let me know). No standard library is used within the firmware.

Based on some rare documentation found from the Internet Archive, the RTOS is actually a micro-kernel which manages tasks, memory allocation and inter-process messaging. Within the kernel, there are two functions that appears to manage memory: fmem_alloc and fmem_free.

I'm trying to locate these functions within the firmware file. I can approximate the location of the code of the RTOS, as well as find functions within that section of code. My goal is to locate the fmem_alloc and fmem_free functions and functions calling them using IDA.

Since my target is a TMS320C55x architecture and I have a raw firmware file versus a neat COFF binary, debugging it is quite difficult without an actual emulator, which from what I understood, is only hardware-based for this microprocessor. As such, it seems I'm doomed to rely only on static analysis. However if anyone knows of a software solution to debug the raw firmware, I would be extremely interested. The Code Composer Studio IDE only debugs COFF binaries.

Is there any techniques, or set of C55x instructions within a certain distance I could search that could identify this function? Or at least reduce the number of potential functions to search? I know the RTOS uses a double linked-list to manage the heap. I first thought of using the malloc function as a potential template, but found it too far from what I'm looking at, i.e. Intel assembly to the C55x, especially given the differences between the Von Neumann and Harvard architectures.

Any help appreciated. Thanks.

P.S. I cannot post product details, firmware files or disassembled code due to non-disclosure conditions. I understand it may be difficult to provide a reliable answer because of this.

Updates

Since my last post, I've identify old Seagate firmware binairies using the same RTOS, but for the Intel 8096. BTW, the target RTOS is an old microkernel called Byte-BOS RTOS, which I wrote about in this post. The only string available in this OS is the copyright notice. However using the Seagate firmware, I can see heap-related error messages. I could potentially try to work these 8096 code segment into C and then recompile them on the TMS320C55x.

Updates 2

So I found out that the compiler/IDE for the platform included somesort of C library which contained a simple malloc() function. I wrote a small C program that allocated some memory with it, ran it in the included debugger and dump the machine code. I made a quick byte string search function in IDAPython and replacing non-instruction values with "????" for IDA to look only for the instructions. Unfortunately, it found no malloc() function. I suspect the code relies exclusively on the memory allocation functions provided by the RTOS rather than the ones from the library. I can hardly believe they would be very different, since both rely on linked list. Anyway, that's where I am with this. I may try to fuzzy the byte string to find code sections that resemble the malloc() function from the library.

Answer 1

I'd approach this problem as follows:

1. Check for logging strings. Probably you'll find there something related to allocations which will solve the problem.

2. Find all functions with 1 argument (size for malloc() and pointer for free()).

3. Track results of these functions. malloc() result is usually checked against NULL after use and rarely used in math operations. In addition result of malloc() function is used in store operations or memcpy().

4. Free function argument is not used after calling it.

5. malloc() and free() should be located in close addresses.

These heuristics should be enough to narrow down the number of possible candidates to observable.

Good luck.

Answer 2

From the TMS320C55x documentation it's clear that the chip supports multiple memory modes. Therefore, a function such as malloc could have different definitions for each one. I think that the smartest thing to do is to identify the memory mode and figure out how a malloc can be implemented in that mode.

Given that your binary file is fully stripped and most functions seem to be inlined, all you can do is pattern matching. If you want to be smart, figure out a way to build a CFG and try to identify where a malloc could be necessary and check if your target pattern is redundant enough.

Hope this helps.