Reverse engineer Listen to this device on Windows - Reverse engineer Windows when there's no API

Question

I'm writing an Autohotkey script to toggle Listen to this device for my microphone, without interacting with a GUI.

I thought it would be a simple registry key being modified so I used RegShot to find the key:

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\MMDevices\Audio\Capture\{My-Microphone's-UUID}\Properties

The key is called {24dbb0fc-9311-4b3d-9cf0-18ff155639d4},1 (On all computers).

And the value when toggling the Listen to this device changes like this: (The 0's change to f's)

xxxxxxxxxxxxxxxx0000xxxx
xxxxxxxxxxxxxxxxffffxxxx

But when I check the GUI, I see that the Listen to this device tick-box has been ticked but I can't actually hear anything from my mic, when I un-tick it, click apply, re-tick it and apply again, I hear my mic. So I thought I might need DllCall or PostMessage here, like what message was sent or what dll was called when I click apply but I couldn't find anything on it on the Internet. I don't know how to make Windows understand that this setting has changed.

Please teach me how to reverse engineer this with x64dbg.

Answer 1

AFAIK, knowing what registry keys are used is not always enough, because the registry is just a place to store things like preferences. Setting a registry value may not have any immediate effect on a device. The real source of truth of how a device is configured is the device itself.

From my experience control panels tend to be lightweight GUIs hosted by the OS, and the process that does any real work is the daemon that the GUI talks to over IPC.

Looking over the recorded API calls in API Monitor we can see that the control panel sends RPC messages to the AudioSrv service. Using sc queryex in Command Prompt, you can find the PID of the svchost (service host) that's hosting the instance of this service.

From there, doing a string search in IDA, we find the string "ListenTo" being used by some of the AudioSrv code. It could be a string used for debugging, but that would be my first place to do some static analysis in IDA or set a breakpoint on with my debugger.

There are some tools that can help with figuring out which code is run when you perform some action. Ultimap in CheatEngine comes to mind, you can find tutorials for it online. You can also perform tracing of the process in x64dbg and look for any syscalls, which is generally interesting because it indicates that the process is asking the kernel for something (e.g. control a device).

Ultimately, this is probably a lot of work just to get to a hacky solution, so you might explore alternatives e.g. creating a virtual device driver.

Answer 2

Unfortunately I cannot answer how to find this information using x64dbg. However this question has been asked multiple times over the years and never got a definitive answer, so here's a solution anyway:

This can be done by acquiring the device via IMMDeviceEnumerator::GetDevice and opening & editing the IMMDevice:PropertyStore directly.

In my experience, IPropertyStore::SetValue will take care of notifying the AudioSrv as well. IPropertyStore::Commit is seemingly not required for events to fire.

Here is a Python sample using comtypes and pycaw:

from comtypes.automation import VT_BOOL
from comtypes.persist import STGM_READWRITE

from pycaw.api.mmdeviceapi.depend import PROPVARIANT
from pycaw.utils import AudioUtilities

CHECKBOX_GUID = "{24DBB0FC-9311-4B3D-9CF0-18FF155639D4}"
CHECKBOX_PID = 1

# Can be found in the registry
DEVICE_GUID = "{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}"

if __name__ == "__main__":
    enumerator = AudioUtilities.GetDeviceEnumerator()
    dev = enumerator.GetDevice(f"{{0.0.1.00000000}}.{DEVICE_GUID}")

    store = dev.OpenPropertyStore(STGM_READWRITE)
    if store is None:
        print("failed to open property store")
        exit(1)

    for j in range(store.GetCount()):
        pk = store.GetAt(j)
        if str(pk.fmtid) != CHECKBOX_GUID or pk.pid != CHECKBOX_PID:
            continue

        new_input = PROPVARIANT(VT_BOOL)
        new_input.union.boolVal = True
        store.SetValue(pk, new_input)
        break

Answer 3

Using nvs' answer as a starting point, I was able to figure out how to also set the device that should be listening to the input.

Here's my code, also using pycaw:

from typing import Optional
from comtypes import GUID
from comtypes.automation import VT_BOOL, VT_LPWSTR, VT_EMPTY
from comtypes.persist import STGM_READWRITE
from pycaw.api.mmdeviceapi import PROPERTYKEY

from pycaw.api.mmdeviceapi.depend import PROPVARIANT
from pycaw.utils import AudioUtilities

#Hardcoded values
LISTEN_SETTING_GUID = "{24DBB0FC-9311-4B3D-9CF0-18FF155639D4}"
CHECKBOX_PID = 1
LISTENING_DEVICE_PID = 0

#Values you can change:
microphone_name = 'Microphone (USB-MIC)'
listening_device_name = 'Speakers (5- GSA 70 Main Audio)'   #Set to None to use the default playback device
enable_listening = False

def main():
    store = get_device_store(microphone_name)
    if store is None:
        print("failed to open property store")
        exit(1)

    set_listening_checkbox(store, enable_listening)
    set_listening_device(store, listening_device_name)

#Write to the checkbox property
def set_listening_checkbox(property_store, value:bool):
    checkbox_pk = PROPERTYKEY()
    checkbox_pk.fmtid = GUID(LISTEN_SETTING_GUID)
    checkbox_pk.pid = CHECKBOX_PID

    new_value = PROPVARIANT(VT_BOOL)
    new_value.union.boolVal = value
    property_store.SetValue(checkbox_pk, new_value)

#Write to the device property
def set_listening_device(property_store, output_device_name:Optional[str]):
    if output_device_name is not None:
        listening_device_guid = get_GUID_from_name(output_device_name)
    else:
        listening_device_guid = None

    device_pk = PROPERTYKEY()
    device_pk.fmtid = GUID(LISTEN_SETTING_GUID)
    device_pk.pid = LISTENING_DEVICE_PID

    if listening_device_guid is not None:
        new_value = PROPVARIANT(VT_LPWSTR)
        new_value.union.pwszVal = listening_device_guid
    else:
        new_value = PROPVARIANT(VT_EMPTY)

    property_store.SetValue(device_pk, new_value)

#Gets the device store from the device name
def get_device_store(device_name:str):
    device_guid = get_GUID_from_name(device_name)
    enumerator = AudioUtilities.GetDeviceEnumerator()
    dev = enumerator.GetDevice(device_guid)

    store = dev.OpenPropertyStore(STGM_READWRITE)
    return store

#This is just a helper method to turn a device name into a GUID.
def get_GUID_from_name(device_name:str) -> str:
    input_devices = get_list_of_active_coreaudio_devices("input")
    for device in input_devices:
        if device.FriendlyName == device_name:
            return device.id
    output_devices = get_list_of_active_coreaudio_devices("output")
    for device in output_devices:
        if device.FriendlyName == device_name:
            return device.id
    raise ValueError("Device not found!")

#Helper method to get all (active) devices
def get_list_of_active_coreaudio_devices(device_type:str) -> list:
    import comtypes
    from pycaw.pycaw import AudioUtilities, IMMDeviceEnumerator, EDataFlow, DEVICE_STATE
    from pycaw.constants import CLSID_MMDeviceEnumerator

    if device_type != "output" and device_type != "input":
        raise ValueError("Invalid audio device type.")

    if device_type == "output":
        EDataFlowValue = EDataFlow.eRender.value
    else:
        EDataFlowValue = EDataFlow.eCapture.value
    # Code to enumerate devices adapted from https://github.com/AndreMiras/pycaw/issues/50#issuecomment-981069603

    devices = list()
    device_enumerator = comtypes.CoCreateInstance(
        CLSID_MMDeviceEnumerator,
        IMMDeviceEnumerator,
        comtypes.CLSCTX_INPROC_SERVER)
    if device_enumerator is None:
        raise ValueError("Couldn't find any devices.")
    collection = device_enumerator.EnumAudioEndpoints(EDataFlowValue, DEVICE_STATE.ACTIVE.value)
    if collection is None:
        raise ValueError("Couldn't find any devices.")

    count = collection.GetCount()
    for i in range(count):
        dev = collection.Item(i)
        if dev is not None:
            if not ": None" in str(AudioUtilities.CreateDevice(dev)):
                devices.append(AudioUtilities.CreateDevice(dev))

    return devices

if __name__ == "__main__":
    main()