IDA Pro for MIPS image

Question

I have this file:

[ytti@lintukoto ~/tmp/ida]% file cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin
cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin: data

It's image for Cisco Catalyst 3650 switch. The switch itself is running Cavium, probably Cavium Octeon CPU on Linux.

I think the file on top of it has some proprietary headers (possible hash of file integrity too). If I remove what I believe to be headers, I get this:

cat3k.mips: mc68010 demand paged executable not stripped

I can't get it open on IDA Pro, as it says it's binary file, and I need to choose entry point.

I'm pretty sure it's self-decompressing image, so I'd probably want to be able to decompress it as well. But at least unzip and gzip fail on me.

What I'm trying to accomplish is, the switch has certain command you use to escape proprietary process and access the Linux shell, before this command executes it presents you with a challenge and you have to give matching response, otherwise no soup^Wshell for you.

I was thinking of finding the string which complains about incorrect response, seeing which part of the code jumps there. I'd expect to find some comparison there, and then I'd just flip the comparison around. With desired results that in future incorrect response would succeed (and correct response fail).
I don't expect any particular efforts from Cisco to avoid customers from breaking their own kit willfully, this is more of easy for them to prove, that customer didn't break it accidentally by going to linux shell and killing some process.

If there is some sort of integrity test of the file, then I might be shit out of luck.

But I'd love some tips how to actually

1. decompress this
2. get it open in IDA Pro in meaningful way

I can share the file upon request (it's useless without the switch, and if you own switch, you are eligible to the image, so I don't consider it 'warez').

The challenge looks like this:

SWITCH#request system shell
Activity within this shell can jeopardize the functioning of the system.
Are you sure you want to continue? [y/n] y
Challenge:
aa1344d4d0dfbf85b22a719e54fe1365b5be267344223bd66c3b6368b3bd2b7f6b8b03fe87ab57e6360634d8fb>586c42c0918a3786f81a3a60
Please enter the shell access response based on the above challenge (Press "Enter" when do>ne or to quit.): 
kakka
Key verification failed

Answer 1

I'd start by plotting the entropy of the file by using for example binwalk which is also part of firmware-mod-kit. This will give you an idea about the layout (bootloader, 2nd stage, main fw etc) of the file and if parts or the whole file is encrypted/compressed. Next you can use the same tool and let it try to automatically find known file types and/or compression algorithm(s) based on signatures.

You can combine these 2 steps in one using

binwalk -B -E <input>

Consult the FAQ/Wiki for more information regarding the usage of binwalk. From that you can start extracting independent blocks and load them into IDA.