Reverse engineering a kernel mode driver (in its 32-bit x86 incarnation) I stumbled over what seems to be an odd calling convention. For a driver I'd expect to see __cdecl, __fastcall and __stdcall in the Microsoft flavor. And since this driver obviously uses its own C++ runtime I'll expect to see __thiscall as well.
However, in this driver I see functions that are passed their first argument in eax. This is completely unexpected, so I am wondering if anyone here has an idea what could be going on?
sub_40XXXX proc near
push ebx
push esi
mov esi, eax
push edi
xor edi, edi
Frame pointer omission doesn't seem like a credible cause for what I am seeing. Is this a bad case of LTCG?
The driver in question is a file system mini filter driver and from the looks of it I'd guess it was linked by the linker in the Windows 7 WDK (although I cannot really say which exact version, e.g. 7600.16385.1 or another). Linker version states 9.00 in the PE optional header. Subsystem version is 5.00 which would indicate that it was built to target Windows 2000. It also suggests that the subsystem was passed explicitly, since the Vista WDKs were the last ones to support targeting Windows 2000. Of course this could also have been built using VS 2008, hard to tell (the standalone WDKs used to include the same optimizing compilers as VS). For all I know it could also be a mix of compilers and linkers from different toolchains. But given that the linker needs to know about calling conventions, I'm still expecting to see some standard calling convention.
Here are the clues for the driver in question (trimmed down version of dumpbin /headers ... output):
9.00 linker version
1000 section alignment
200 file alignment
5.00 operating system version
0.00 image version
5.00 subsystem version
The standard calling convention for drivers is __stdcall. But to verify that __fastcall doesn't indeed use eax as I see in the other driver, I decided to create a little driver. In order to prevent the compiler or linker to optimize out my function calls, I messed up some variables subsequently passed to IoCreateSymbolicLink.
The C++ code for the respective two functions is:
UINT_PTR __fastcall fastcall_test(UINT_PTR arg1, UINT_PTR arg2)
{
UINT_PTR ret = arg1 + arg2;
DbgPrint("%u, %u -> %u", arg1, arg2, ret);
return ret;
}
UINT_PTR __stdcall stdcall_test(UINT_PTR arg1, UINT_PTR arg2)
{
UINT_PTR ret = arg1 + arg2;
DbgPrint("%u, %u -> %u", arg1, arg2, ret);
return ret;
}
and they're called as:
UINT_PTR fct = fastcall_test((UINT_PTR)status, RegistryPath->MaximumLength);
UINT_PTR sct = stdcall_test((UINT_PTR)status, RegistryPath->MaximumLength);
usSymlinkName.Buffer += fct;
usSymlinkName.Length += (USHORT)fct;
usSymlinkName.Buffer += sct;
usSymlinkName.MaximumLength += (USHORT)(sct + fct);
from DriverEntry between IoCreateDevice and IoCreateSymbolicLink in a default project generated using DDKWizard.
When compiling this targeting Windows XP, the outcome is as follows:
.text:00010512 unsigned int __fastcall fastcall_test(unsigned int, unsigned int) proc near
.text:00010512 ; CODE XREF: DriverEntry(x,x)+41p
.text:00010512 arg1 = ecx
.text:00010512 arg2 = edx
.text:00010512 mov edi, edi
.text:00010514 push esi
.text:00010515 lea esi, [arg1+arg2]
.text:00010518 push esi
.text:00010519 push arg2
.text:0001051A push arg1
.text:0001051B push offset Format ; "%u, %u -> %u"
.text:00010520 call _DbgPrint
.text:00010525 add esp, 10h
.text:00010528 mov eax, esi
.text:0001052A pop esi
.text:0001052B retn
.text:0001052B unsigned int __fastcall fastcall_test(unsigned int, unsigned int) endp
.text:00010532 unsigned int __stdcall stdcall_test(unsigned int, unsigned int) proc near
.text:00010532 ; CODE XREF: DriverEntry(x,x)+50p
.text:00010532
.text:00010532 arg1 = dword ptr 8
.text:00010532 arg2 = dword ptr 0Ch
.text:00010532
.text:00010532 mov edi, edi
.text:00010534 push ebp
.text:00010535 mov ebp, esp
.text:00010537 mov eax, [ebp+arg1]
.text:0001053A mov ecx, [ebp+arg2]
.text:0001053D push esi
.text:0001053E lea esi, [eax+ecx]
.text:00010541 push esi
.text:00010542 push ecx
.text:00010543 push eax
.text:00010544 push offset Format ; "%u, %u -> %u"
.text:00010549 call _DbgPrint
.text:0001054E add esp, 10h
.text:00010551 mov eax, esi
.text:00010553 pop esi
.text:00010554 pop ebp
.text:00010555 retn 8
.text:00010555 unsigned int __stdcall stdcall_test(unsigned int, unsigned int) endp
As expected __fastcall ends up passing arguments via ecx and edx. So what's going on with my other driver?
Meanwhile I found out how the PE header values could be achieved using a vanilla Windows 7 WDK. This will then yield a binary which is compatible with Windows 2000, assuming you build for x86, and contain these exact values (and of course assuming you don't do stupid things like statically importing DDIs only available after Windows 2000).
In sources specify ...
USE_MAKEFILE_INC=1
SUBSYSTEM_VERSION=$(SUBSYSTEM_500)
# Alternatively:
#SUBSYSTEM_VERSION=5.00
which will force nmake to include makefile.inc from the same location as the sources file and set the SUBSYSTEM_VERSION correctly (5.00 for x86 and 5.02 for amd64).
Then in makefile.inc override
LINKER_APP_VERSION=0.00
LINKER_OS_VERSION=$(SUBSYSTEM_VERSION)
which works because makefile.inc gets included quite late from makefile.new and therefore we can use it to override the defaults specified by the default build environment.
