I recently started reverse engineering a firmware from a device for which I had not documentation other than the interfaces to it. The presentation provided in the previous answer is very helpful. I thought I would complement it with some of my experience from my current project.
My first step in any RE project is to extract the strings - both ASCII and Unicode. Sometimes,
strings or IDA may not find everything, you may need to conduct a review of the hex code as some programs may have some uncommon encoding mechanism. While the presence of string may not provide you much in some cases, the absence of any string is a strong indication of compression or encryption. In my case, I was able to retrieve copyright notices containing information about some internal programs, along with a copyright notice for the real-time operating system included. Most helpful.
Be careful with
binwalk, which will often provide false positive for uncommon firmware. Don't rely on its output.
Many firmware will have a similar structure, i.e. some initialization, maybe an interrupt vector with a RESET interrupt at the beginning, then eventually jump to a bootstrapping section, which will load further components into memory. I found the bin2bmp tool useful to provide an overview of the contents. Note: if you use this tool in Windows, you will need the Python Imaging Library
Additionally, each program within the firmware will have sections for code and data at least. The code section will be much larger than the data section, and from my experience, precedes the data section. Combined with the
bin2bmp tool, you can start identifying code sections. Additional sections for heap, static variables and exception handling may be added.
Most developers will use C/C++ as their language, sometimes Assembly. When C/C++ is used, you can safely assume some instructions to occur fairly often. The presentation from Recon mention the
RET instruction, which I found true. Furthermore, the function epilogs and prologs are often similar across the entire firmware. If you can have some statistics about the count of the instructions, you may be able to identify a particular byte/word/dword to the
RET instruction. Afterwards, observe the 2-3 previous bytes/words and verify if they reoccur together across the code section, indicating you may have found the epilog of functions. You can use IDA in plain binary mode to search byte strings easily. Prologs of functions will often consist of a
POP instruction to store the returning address or load arguments. The epilog usually contains a
PUSH instruction prior to the
RET. You can then attempt to locate online for instruction sets which have a
PUSH instruction corresponding to the byte/words you have identified. Consider endianness when analyzing the binary.
I'm including these documents I found useful on my project, which may help you as well:
I'll be monitoring that question, as I'm eager to see what else can be done to understand firmwares with uncommon architectures.