I'm currently working on reversing a firmware binary for an embedded device. This device is using an uncommon architecture: it is a modified Harvard architecture, running on a TMS320C55x chipset with an old Real-time Operating System (RTOS) which does not exists anymore (Byte-BOS) and for which I cannot find any manual (if you have, please let me know). No standard library is used within the firmware.
Based on some rare documentation found from the Internet Archive, the RTOS is actually a micro-kernel which manages tasks, memory allocation and inter-process messaging. Within the kernel, there are two functions that appears to manage memory:
I'm trying to locate these functions within the firmware file. I can approximate the location of the code of the RTOS, as well as find functions within that section of code. My goal is to locate the
fmem_free functions and functions calling them using IDA.
Since my target is a TMS320C55x architecture and I have a raw firmware file versus a neat COFF binary, debugging it is quite difficult without an actual emulator, which from what I understood, is only hardware-based for this microprocessor. As such, it seems I'm doomed to rely only on static analysis. However if anyone knows of a software solution to debug the raw firmware, I would be extremely interested. The Code Composer Studio IDE only debugs COFF binaries.
Is there any techniques, or set of C55x instructions within a certain distance I could search that could identify this function? Or at least reduce the number of potential functions to search? I know the RTOS uses a double linked-list to manage the heap. I first thought of using the
malloc function as a potential template, but found it too far from what I'm looking at, i.e. Intel assembly to the C55x, especially given the differences between the Von Neumann and Harvard architectures.
Any help appreciated. Thanks.
P.S. I cannot post product details, firmware files or disassembled code due to non-disclosure conditions. I understand it may be difficult to provide a reliable answer because of this.
Since my last post, I've identify old Seagate firmware binairies using the same RTOS, but for the Intel 8096. BTW, the target RTOS is an old microkernel called Byte-BOS RTOS, which I wrote about in this post. The only string available in this OS is the copyright notice. However using the Seagate firmware, I can see heap-related error messages. I could potentially try to work these 8096 code segment into C and then recompile them on the TMS320C55x.
So I found out that the compiler/IDE for the platform included somesort of C library which contained a simple malloc() function. I wrote a small C program that allocated some memory with it, ran it in the included debugger and dump the machine code. I made a quick byte string search function in IDAPython and replacing non-instruction values with "????" for IDA to look only for the instructions. Unfortunately, it found no malloc() function. I suspect the code relies exclusively on the memory allocation functions provided by the RTOS rather than the ones from the library. I can hardly believe they would be very different, since both rely on linked list. Anyway, that's where I am with this. I may try to fuzzy the byte string to find code sections that resemble the malloc() function from the library.