5

I have several Solaris 2.6 era drivers I would like to reverse engineer.

I have a Sparc disassembler which provides some info but it isn't maintained anymore so I think it may not give me all the information possible.

The drivers are for an Sbus graphics accelerator I own. Namely the ZX aka Leo one of the early 3d accelerators.

So what are some ways I can go about reverse engineering this driver? I can disassemble it but I am not sure what to make of the output. I also have Solaris of course so perhaps there are things I can do there as well.

The final goal is to have enough information to design a driver for an Operating System. There are drivers for NetBSD, although incomplete as the hardware documentation that does exist (isn't free to access) does not have the Window ID encoding as it is missing. Also, since the hardware uses an Sbus interface on a double wide mezzanine card, it would be impractical to use it on anything but a SparcStation or early UltraSparc machine.

  • 2
    You probably should add what is your final goal. Do you want to use the card in another machine? Make a driver for another OS on the same box? Just curiosity? – Igor Skochinsky Mar 20 '13 at 2:58
  • @Igor Skochinsky Good suggestion I have taken you up on it! And yes just for curiosity and fun.. Vintage SparcStations are my hobby you might say. – cb88 Mar 20 '13 at 12:41
  • If you have drivers for NetBSD I'd recommend looking at them and trying to find matches with what the Solaris driver does. This should speed up the process. – Igor Skochinsky Mar 20 '13 at 15:31
  • Indeed however as I said...the NetBsd driver for the ZX is incomplete and it certainly doesn't cover the opengl 1.1 capability that the device has so I'll almost definitly have to get a copy of IDA or have a friend dump the assembly for me. I also have a TGS opengl driver for it that would be intersting I am sure however it requires a flexlm license which sadly I do not have. – cb88 Mar 20 '13 at 15:48
  • I also found some kind of a Linux driver that might be useful. – Igor Skochinsky Mar 20 '13 at 16:22
10

Well, since it's a Solaris driver, first you need to find up some docs on how Solaris drivers communicate with the kernel (or kernel with them). A quick search turned up this:

_init() initializes a loadable module. It is called before any other routine in a loadable module. _init() returns the value returned by mod_install(9F) . The module may optionally perform some other work before the mod_install(9F) call is performed. If the module has done some setup before the mod_install(9F) function is called, then it should be prepared to undo that setup if mod_install(9F) returns an error.

_info() returns information about a loadable module. _info() returns the value returned by mod_info(9F).

_fini() prepares a loadable module for unloading. It is called when the system wants to unload a module. If the module determines that it can be unloaded, then _fini() returns the value returned by mod_remove(9F). Upon successful return from _fini() no other routine in the module will be called before _init() is called.

There's a nice code sample below.

This guide also seems relevant.

Once you found the entry points, it's just a matter of following the calls and pointers.

Here's how it looks in IDA:

.text:00000000 _init:                                  ! DATA XREF: leo_attach+5A8o
.text:00000000                                         ! leo_attach+5BCo ...
.text:00000000                 save    %sp, -0x60, %sp
.text:00000004                 sethi   %hi(leo_debug), %i2
.text:00000008                 ld      [leo_debug], %o0
.text:0000000C                 cmp     %o0, 4
.text:00000010                 bl      loc_38
.text:00000014                 sethi   %hi(leo_state), %o0
.text:00000018                 set     aLeoCompiledSS, %o0 ! "leo: compiled %s, %s\n"
.text:00000020                 set     a141746, %o1    ! "14:17:46"
.text:00000028                 sethi   %hi(aLeo_c6_6Jun251), %l0 ! "leo.c 6.6 Jun 25 1997 14:17:46"
.text:0000002C                 call    leo_printf
.text:00000030                 set     aJun251997, %o2 ! "Jun 25 1997"
.text:00000034                 sethi   %hi(leo_state), %o0
.text:00000038
.text:00000038 loc_38:                                 ! CODE XREF: _init+10j
.text:00000038                 set     leo_state, %i1
.text:0000003C                 sethi   %hi(0x1800), %l0
.text:00000040                 mov     %i1, %o0
.text:00000044                 set     0x1980, %o1
.text:00000048                 call    ddi_soft_state_init
.text:0000004C                 mov     1, %o2
.text:00000050                 orcc    %g0, %o0, %i0
.text:00000054                 bne,a   loc_80
.text:00000058                 ld      [%i2+(leo_debug & 0x3FF)], %o0
.text:0000005C                 sethi   %hi(0x14C00), %l0
.text:00000060                 call    mod_install
.text:00000064                 set     modlinkage, %o0
.text:00000068                 orcc    %g0, %o0, %i0
.text:0000006C                 be,a    loc_80
.text:00000070                 ld      [%i2+(leo_debug & 0x3FF)], %o0
.text:00000074                 call    ddi_soft_state_fini
.text:00000078                 mov     %i1, %o0
.text:0000007C                 ld      [%i2+(leo_debug & 0x3FF)], %o0
.text:00000080
.text:00000080 loc_80:                                 ! CODE XREF: _init+54j
.text:00000080                                         ! _init+6Cj
.text:00000080                 cmp     %o0, 4
.text:00000084                 bl      locret_9C
.text:00000088                 nop
.text:0000008C                 set     aLeo_initDoneRe, %o0 ! "leo: _init done, return(%d)\n"
.text:00000094                 call    leo_printf
.text:00000098                 mov     %i0, %o1
.text:0000009C
.text:0000009C locret_9C:                              ! CODE XREF: _init+84j
.text:0000009C                 ret
.text:000000A0                 restore
.text:000000A0 ! End of function _init

At 0x60 you can see mod_install being called with a pointer to modlinkage, so you can follow there and see what the fields are pointing to.

But you don't even have to do that all the time. In this case, the programmers very thoughtfully left intact all the symbols and debug output. This should help you in your work :)

Depending on situation, you may skip straight to the helpfully-named functions like leo_blit_sync_start or leo_init_ramdac. I personally prefer the first way, top-down, but to each his own.

EDIT: one rather simple thing you can do is to patch the leo_debug variable at the start of .data section to 5 or so. That should produce a lot of debug output about the operations the driver is performing.

| improve this answer | |
  • Clearly IDA pro speaks for itself :) ... that said I can't afford a license for now. It is a bit off topic but per architecture licensing might be interesting for people like myself. – cb88 Mar 20 '13 at 13:04
  • @cb88: I reckon it'd still be more expensive than you will like. IDA is unique. The price is fair, but not to everyone. They have student licenses, however. But I think you need someone to know you and vouch for you (if they still do that). In the past too many leaks have caused financial damage to DataRescue and later Hex-Rays. – 0xC0000022L Apr 4 '13 at 17:35
  • 0xC0000022L $75 would be 3x what they charge per arch in the full software and if you could later upgrade it to the full version for the full price - the single arch price that would be quite fair but I suppose it is too much to ask. I don't doubt its worth every pennny I just don't need a battleship for my paddle boat ride of a job :D – cb88 Apr 4 '13 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.