You may want to start with something simple like monitoring the API calls the controller application is doing. I'm not sure how old your Windows version is but it might be worth giving API Monitor a try. It gives you a good start to monitoring the application at least and you get to trivially see how it interacts with the operating system.
Essentially what you're looking for is probably CreateFile followed by DeviceIoControl calls which use the same handle as returned from CreateFile. This is one way for an application to interact with kernel mode software. Another option is to look for a CreateFile call followed by ReadFile and WriteFile which is another common way for user space applications to communicate with kernel mode drivers.
If your software is so old that we're talking Windows 9x you should probably use something like SoftICE.
Once you've figured out these things you'd need to examine the driver for the card and look for the handlers for the DeviceIoControl calls you saw before or the ReadFile, WriteFile handling. We've had a similar discussion here and there is a decent resource available as Kernel Mode Driver Tutorial by Clandestiny which is getting a bit long in the tooth but still applies.
This is quite a bit of work and I would strongly advice you to buy something like Practical Reverse Engineering, as well as A bug hunter's diary. They both discuss the issues of how to reverse engineer user space and kernel mode interaction. If you're very serious about this I would recommend also getting a copy of Windows Internals, Part 1 and Windows Internals, Part 2.
If you want to look at the actual PCI traffic, which is probably not what you want to do, I think you have to go the hardware route. There's a PCI bus analyzer available from Silicon Control, then there's the expensive solutions from the big names, LeCroy and Agilent. Although I'm not sure if they still make PCI bus analyzers anymore or if everyone has moved on to PCIe. You also have the option of breaking out the bus yourself and using an FPGA to sniff the signals. ElectroFriends has a short introduction to the PCI bus available.