2

I am disassembling a legacy 32-bit (x86) application using IDA Home (7.6) (for x86 disassembly) in a 64-bit Windows 10 environment running in a virtual machine (VMWare Fusion for Mac). The application executes without issue when outside a debugger, however, when running the application using the "Local Windows Debugger" in IDA (loaded as a MetaPC project), an "unknown exception code 6A6" is thrown:

                                        IDA Error Exception Message

This exception occurs shortly after the application execution has begun in the debugger (i.e. without any application user input).

To eliminate a possible issue with IDA, I tested using the x32dbg debugger and the Ollydbg debugger. Both debuggers throw the same raised exception, namely the RPC_S_INVALID_BINDING (0x000006A6) error:

                        x32dbg Raised Exception

I spent some time thinking about why I am facing issues running the application inside a debugger and I came up with a few possible reasons:

  1. Anti-debugger techniques.
  2. 64-bit debuggers are loading the wrong DLLs (i.e. they're loading the 64-bit DLL when it should be loading the 32-bit DLL located in C:\WINDOWS\SysWOW64\).
  3. Something else (??)

I am able to disprove that both #1 and #2 are not the issues after setting up a 64-bit Windows 7 with Service Pack 1 environment. I am able to debug the application in IDA without issue while in this Windows 7 environment (i.e. the 6A6 exception is not thrown).

So, this leads me to #3 - a head scratching moment. I did some research into the RPC_S_INVALID_BINDING (0x000006A6) error and came across this StackOverflow question. Naming, the accepted answer notes the following:

Explaining 'The Binding Handle Is Invalid'

Today I want to try to give more insight into the 'Binding Handle Is Invalid' problem that a number of people have reported with the VS 2005 debugger.

First, if all you care about is how to solve the problem: Enable the 'Terminal Services' service and reboot. If you want to know more, read on.

I tried enabling the Terminal Services (now named Remote Desktop Services on Windows 10 [reference]), however, this doesn't solve the issue and I still get an "unknown exception code 6A6" error. The RPC_S_INVALID_BINDING (0x000006A6) error confuses me quite a bit.

There clearly appears to be something wrong with my environment setup with Windows 10 that isn't allowing me to specifically debug this application. Something changed from Windows 7 that is causing this issue and I'm at a loss here.

Does anyone know what might be the issue here?


Environment Detail(s):

  • OSes Tested in VMWare Fusion (12.2.1 (18811640)):
    • [❌ - Issue] Windows 10 (x64) - Build 19042 (20H2 / 20H2)
    • [❌ - Issue] Windows 10 (x64) - Build 15063 (1703 / Redstone 2)
    • [❌ - Issue] Windows 10 (x64) - Build 14393 (1607 / Redstone 1)
    • [✅ - No Issue] Windows 7 Ultimate (x64) - Build 7601 (Service Pack 1)
  • Tools Used (launched as Administrator):
    • IDA Home 7.6 (v7.6.210427) (x64)
    • x32dbg (Release "July 1 2021, 23:09:08") (x64)
    • Ollydbg (v2.01) (x86)

Update #1:

As requested, here is the call stack (a.k.a. stack trace) when the exception occurs:

Call stack at the point of the exception

The application appears to be checking the DirectX9 environment and whether it is supported or not. Although, this is a guess on my part from the name of the DLL, DX9EnvChk.dll, which is included alongside the application in question. I can't find much information about this DLL through a normal web search, but it appears to be present with other applications of the era. This DLL (nor accompanying header file) was not part of the Microsoft DirectX SDK (June 2010) from what I can see.

4
  • Hi and welcome to RE.SE. Have you looked at what shims get applied? You stress the fact it's a legacy application and we already know it would get to run inside WOW64 on all your tested OSs. Other things I'd look at are 1.) anti-malware solutions, 2.) active mitigations 3.) the event log (oftentimes RPC and COM issues will be logged). Also, have you looked at the call stack at the time of the exception?
    – 0xC0000022L
    Dec 17 '21 at 9:59
  • @0xC0000022L - There are no custom shims installed along side this application. I verified no custom shims were applied by checking HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\InstalledSDB - at the time of installation, this key didn't exist. The app likewise doesn't need to be run in a compatibility mode (when running outside a debugger). I also checked the Event Viewer logs for any RPC/COM logs (3), but nothing is logged. Lastly, I just updated my answer with the call stack. The exception in the debugger appears to be happening during the DirectX9 environment check.
    – Code Doggo
    Dec 19 '21 at 3:44
  • @0xC0000022L - Can you provide some more insights on #1 and #2 you listed as "things [to] look at?" Let me also know if you think of any more insight from the updated information I provided in my comment above. I should also note that I am running these OSes in a virtual machine (i.e. VMWare Fusion on Mac). I updated my post above to reflect this.
    – Code Doggo
    Dec 19 '21 at 3:49
  • Thanks for the update. Question 1 was geared at solutions that try to preempt exploits (and since Windows 8 or so more often than not subvert the built-in mechanisms WIndows has gained). If you use Windows Defender, you probably needn't care. Active mitigations come in a number of shapes. I suggest you use Process Explorer and look at the various columns you can activate in the main view as well as the Properties dialog for the process. It's unlikely that mitigations are activated for an old executable image, though. How many functions does the DX9EnvChk.dll export?
    – 0xC0000022L
    Dec 19 '21 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.