as I understand your question, if your binary doesn't have a packer / protector, and you can't use any way to hook the original source, try using a kernel hooke like TitanHide, and observe the behavior, now if not having anything so complicated check the sides of your debugger look for flags and find the "zero flag", and change your return, this is ...
I can think of the following options:
a 1-bit debug output (e.g. a LED, one-way output-only serial, or something else). probing it with an oscilloscope or a logic analyzer should give more clues.
it's a jumper/strap pin which puts the device into some kind of a debug or recovery mode if shorted.
I think you're not doing anything wrong, you're experiencing this bug:
Usually if you retry several times you might get it to work eventually; it looks like a race condition.
The patch proposed in the bug report kind of works for me on macOS 10.15.5: instead of just hanging, GDB errors out upon run with:
ActiveX controls (.ocx) are actually simple DLLs conforming to the COM standard. In practice, it means that it exports standard COM entry points such as:
The most important for us is DllGetClassObject as this is the one called by the host process to create the ActiveX control(s).