Reverse engineering a Solaris driver

Question

I have several Solaris 2.6 era drivers I would like to reverse engineer.

I have a Sparc disassembler which provides some info but it isn't maintained anymore so I think it may not give me all the information possible.

The drivers are for an Sbus graphics accelerator I own. Namely the ZX aka Leo one of the early 3d accelerators.

So what are some ways I can go about reverse engineering this driver? I can disassemble it but I am not sure what to make of the output. I also have Solaris of course so perhaps there are things I can do there as well.

The final goal is to have enough information to design a driver for an Operating System. There are drivers for NetBSD, although incomplete as the hardware documentation that does exist (isn't free to access) does not have the Window ID encoding as it is missing. Also, since the hardware uses an Sbus interface on a double wide mezzanine card, it would be impractical to use it on anything but a SparcStation or early UltraSparc machine.

Answer 1

Well, since it's a Solaris driver, first you need to find up some docs on how Solaris drivers communicate with the kernel (or kernel with them). A quick search turned up this:

_init() initializes a loadable module. It is called before any other routine in a loadable module. _init() returns the value returned by mod_install(9F) . The module may optionally perform some other work before the mod_install(9F) call is performed. If the module has done some setup before the mod_install(9F) function is called, then it should be prepared to undo that setup if mod_install(9F) returns an error.

_info() returns information about a loadable module. _info() returns the value returned by mod_info(9F).

_fini() prepares a loadable module for unloading. It is called when the system wants to unload a module. If the module determines that it can be unloaded, then _fini() returns the value returned by mod_remove(9F). Upon successful return from _fini() no other routine in the module will be called before _init() is called.

There's a nice code sample below.

This guide also seems relevant.

Once you found the entry points, it's just a matter of following the calls and pointers.

Here's how it looks in IDA:

.text:00000000 _init:                                  ! DATA XREF: leo_attach+5A8o
.text:00000000                                         ! leo_attach+5BCo ...
.text:00000000                 save    %sp, -0x60, %sp
.text:00000004                 sethi   %hi(leo_debug), %i2
.text:00000008                 ld      [leo_debug], %o0
.text:0000000C                 cmp     %o0, 4
.text:00000010                 bl      loc_38
.text:00000014                 sethi   %hi(leo_state), %o0
.text:00000018                 set     aLeoCompiledSS, %o0 ! "leo: compiled %s, %s\n"
.text:00000020                 set     a141746, %o1    ! "14:17:46"
.text:00000028                 sethi   %hi(aLeo_c6_6Jun251), %l0 ! "leo.c 6.6 Jun 25 1997 14:17:46"
.text:0000002C                 call    leo_printf
.text:00000030                 set     aJun251997, %o2 ! "Jun 25 1997"
.text:00000034                 sethi   %hi(leo_state), %o0
.text:00000038
.text:00000038 loc_38:                                 ! CODE XREF: _init+10j
.text:00000038                 set     leo_state, %i1
.text:0000003C                 sethi   %hi(0x1800), %l0
.text:00000040                 mov     %i1, %o0
.text:00000044                 set     0x1980, %o1
.text:00000048                 call    ddi_soft_state_init
.text:0000004C                 mov     1, %o2
.text:00000050                 orcc    %g0, %o0, %i0
.text:00000054                 bne,a   loc_80
.text:00000058                 ld      [%i2+(leo_debug & 0x3FF)], %o0
.text:0000005C                 sethi   %hi(0x14C00), %l0
.text:00000060                 call    mod_install
.text:00000064                 set     modlinkage, %o0
.text:00000068                 orcc    %g0, %o0, %i0
.text:0000006C                 be,a    loc_80
.text:00000070                 ld      [%i2+(leo_debug & 0x3FF)], %o0
.text:00000074                 call    ddi_soft_state_fini
.text:00000078                 mov     %i1, %o0
.text:0000007C                 ld      [%i2+(leo_debug & 0x3FF)], %o0
.text:00000080
.text:00000080 loc_80:                                 ! CODE XREF: _init+54j
.text:00000080                                         ! _init+6Cj
.text:00000080                 cmp     %o0, 4
.text:00000084                 bl      locret_9C
.text:00000088                 nop
.text:0000008C                 set     aLeo_initDoneRe, %o0 ! "leo: _init done, return(%d)\n"
.text:00000094                 call    leo_printf
.text:00000098                 mov     %i0, %o1
.text:0000009C
.text:0000009C locret_9C:                              ! CODE XREF: _init+84j
.text:0000009C                 ret
.text:000000A0                 restore
.text:000000A0 ! End of function _init

At 0x60 you can see mod_install being called with a pointer to modlinkage, so you can follow there and see what the fields are pointing to.

But you don't even have to do that all the time. In this case, the programmers very thoughtfully left intact all the symbols and debug output. This should help you in your work :)

Depending on situation, you may skip straight to the helpfully-named functions like leo_blit_sync_start or leo_init_ramdac. I personally prefer the first way, top-down, but to each his own.

EDIT: one rather simple thing you can do is to patch the leo_debug variable at the start of .data section to 5 or so. That should produce a lot of debug output about the operations the driver is performing.