Disassemble communication protocol for an old device

Question

I have an old device connected to personal computer via specific PCI card. Device is handled with C++ control application, which is not able to run on new versions of Windows. Manufacturer of that device was consumed by big company a while ago and do not continue on development of such devices.

What I want to do is disassemble communication protocol between this device and computer, respective between the software and PCI card; however, I am complete beginner. I downloaded IDA tool. I am able to trace application and find couple of subroutines that are often trigerred when application is sending commands to the device, but these subroutines contains others etc etc. Also I could not find any meaningful strings in the disassembled application (i was expecting many short string instructions with numeric parameters).

I would like to ask you for some advice, what I should do at first, or what to be careful on when I want to detect communication protocol.

Answer 1

You may want to start with something simple like monitoring the API calls the controller application is doing. I'm not sure how old your Windows version is but it might be worth giving API Monitor a try. It gives you a good start to monitoring the application at least and you get to trivially see how it interacts with the operating system.

Essentially what you're looking for is probably CreateFile followed by DeviceIoControl calls which use the same handle as returned from CreateFile. This is one way for an application to interact with kernel mode software. Another option is to look for a CreateFile call followed by ReadFile and WriteFile which is another common way for user space applications to communicate with kernel mode drivers.

If your software is so old that we're talking Windows 9x you should probably use something like SoftICE.

Once you've figured out these things you'd need to examine the driver for the card and look for the handlers for the DeviceIoControl calls you saw before or the ReadFile, WriteFile handling. We've had a similar discussion here and there is a decent resource available as Kernel Mode Driver Tutorial by Clandestiny which is getting a bit long in the tooth but still applies.

This is quite a bit of work and I would strongly advice you to buy something like Practical Reverse Engineering, as well as A bug hunter's diary. They both discuss the issues of how to reverse engineer user space and kernel mode interaction. If you're very serious about this I would recommend also getting a copy of Windows Internals, Part 1 and Windows Internals, Part 2.

If you want to look at the actual PCI traffic, which is probably not what you want to do, I think you have to go the hardware route. There's a PCI bus analyzer available from Silicon Control, then there's the expensive solutions from the big names, LeCroy and Agilent. Although I'm not sure if they still make PCI bus analyzers anymore or if everyone has moved on to PCIe. You also have the option of breaking out the bus yourself and using an FPGA to sniff the signals. ElectroFriends has a short introduction to the PCI bus available.