It depends on the particular interface, but the general idea is often the same and usually does NOT involve disassembling the driver. The reason is that the disassembling method for reverse engineering is not considered legal in all jurisdictions. Here are the usual steps:
1. Contact the manufacturer for needed information
Obviously, the most direct approach is also the easiest if successful. Some manufacturers are more cooperative than others. For some types of devices, even if the manufacturer of a device is uncooperative, sometimes the manufacturer of major chips on the device are willing to share some information and this can help.
2. Get general information about the device by observation
A lot of information can be obtained simply by observation. For example, what are the major components in the device? Can we look up part numbers and find datasheets for them? Is this version n+1 and there is already a version n driver for Linux? Does the manual or the Windows driver user interface give any important clues about registers, settings or capabilities? Does the Windows driver support multiple devices? This can be an indication that the devices might be similar, and if there's a Linux driver for one of them already, it can help.
3. Capture communications with the device for analysis
For some devices such as serial ports or USB, capturing the communications between the device driver and device is usually fairly straightforward. (See How to reverse engineer simple usb device [windows -> linux]) Capturing communications for video cards can be done in a couple of ways. One way it can be done is by using a proprietary Linux driver and then intercepting calls for that, as with the REnouveau tool. Sharing data is important, and is one reason so many drivers have been successfully written for Linux. One of the major strengths of the open source community is the fact that there are people all over the world who can and do collaborate with such efforts.
4. Attempt to duplicate the communications under Linux
This is a matter of writing code and trying the result. Because Linux already has a rich set of drivers, it's often easiest to start with something similar and modify it. Since everything is open source, tinkering is not merely allowed but encouraged! For devices other than video cards, one can often write userspace code to attempt to exercise the device and to gather data and try experiments. Ultimately, a real driver is written, ideally as a kernel module, to allow unloading and reloading. This speeds the development cycle over having to reboot after any driver change (I'm looking at YOU, Windows!)
5. Test extensively
Testing and bugfixing is important for open source software generally, and that certainly applies to device drivers as well as anything else. First, for inclusion in the kernel, other developers look at the code in a sometimes brutal but invariably useful process called code review. Other experienced developers look at the source code and point out weaknesses in the code, flaws in assumptions and even typos in comments and formatting problems. Once enough people have done that, people with the actual device in question actually try it on their hardware and report bugs or anomalies. This often turns up issues such as version differences (e.g. there were multiple versions of the physical device, but all sold as the same thing) and device conflicts (both device A and device B work great individually, but fail when both are connected).
Ultimately, the result is a shiny new bug-free open source driver that everyone can use. (Or that's the goal, anyway!)
