C++, Windows UI Automation and DaVinci Resolve

Question

I've been researching this topic for a while, but the information is extremely limited and very confusing.

DaVinci Resolve is an editing and color grading software, its tools are so many and so specialized that external hardware has been created to speed up the workflow.

Now, such tools are crazy expensive, so I wanted to create my own version with a microcontroller. The microcontroller programming isn't the hard part, making it interact with the UI is.

I have tried Cheat Engine to check the parameters in memory, and that worked, however I can write a bit of C++ but definitely not assembly, also manually changing values in a memory address during execution probably isn't the right way of doing it, so I was looking for another solution: UI automation.

It's crazy how there is no API documentation for Microsoft's UIA, and if there is, they've done a great job at hiding it. I just don't know how to start writing code for it.

Also, the "Inspector" tool in Windows shows that the parameters in the UI have no value, but rather the value is written in the name of the parameter.

I know that DaVinci Resolve is written using QT5, I tried looking for documentation about it, but it took me to 3rd party paid software and that's not what I need.

I have the most experience with Python, and I did try pywinauto. I managed to attach to the process and print the entire UI tree. The problem is that 99% of the elements show up as "Custom" elements without name, value, or any kind of control.

With all of this considered, I would like some help on how to approach programmatically control a 3rd party software.

Answer 1

Qt does UI rendering on its own instead of relying on standard Win32 APIs, so accessibility APIs rarely work with Qt apps unless they implemented necessary interfaces. You may have better luck hooking Qt APIs inside the app. This could be a start:

https://github.com/robertknight/Qt-Inspector

See also this article, although it’s likely outdated.

Answer 2

For Windows environments, you can easily use WinAppDriver to manipulate UI elements. It relies on the mentioned paths, and heavily on XPATHs format.

WinAppDriver works like a server, making the manipulation of UI elements (click, write, read, select, etc) easily by JSON messages. Several thirth party programs help doing that creating JSON messages with requests for the kind of manipulation and simplifing the use of the server.

Client application that wants to manipulate the UI connect via socket to the WinAppDriver port and send command, as mentioned, to click, move mouse, type like keyboard. Example client applications:

• WinUIRecorder, used to identify buttons and dialog boxes path
• Appium to automate tasks in a more friendly UI
• RobotFramework together with WADLibrary to easily write natural language UI manipulation scripts.

All these work on top of the socket as clients. After you learn how the socket communication is estabilished and requests are made, you can easily create your own requests in the language of your choice. The requests are all in JSON, easily to be understand by anyone with basic programmming knowledge.

WinAppDriver works fine on Windows 10 or above. I already ised on UWP apps, WPF, and other applications. This kind of UI manipulation in client/server, with UI elements mapped via XPATHs are available on other platforms too, like Android and Apple, to help automate tasks and testing.

Microsoft also release in 2021 or 2022 a tool to help automate tasks, based on the same infrastructure, but I can recall the name of it.