IDA/Hexrays: local variable allocation has failed; how to fix this?

Question

I'm reverse engineering an old DOS game that's using the DOS/4GW extender. The game includes some libraries that I already successfully reversed in a different application, so I'm pretty sure that the declaration for the function that is making trouble is correct. Nonetheless, the function in question shows the error message // local variable allocation has failed, the output may be wrong!. When looking at the assembly, it seems like there's a problem with the stack definition, as can be seen here:

I already tried to edit the function and make some adaptions (unfortunately I'm lacking a little bit of experience there), but to no avail. Here's a screenshot of the declarations/stack frame and the function's initial options without changes:

What could be the cause for the failed allocation and how can one fix it?

Answer 1

First, local variable allocation failures do not necessarily mean that the decompilation is incorrect. They can happen naturally when using forced variables, for example. I'd proceed trying to analyze the function as normal.

If you see any weird variable manipulations that you can't reconcile, I would recommend double-checking the stack pointer all throughout the function. Start with the return sites: do you see any negative displacements at ret instructions? If so, this indicates wrong stack points somewhere throughout the function.

If the return sites look good, I would recommend pulling up View->Open subviews->Problems and get to work fixing anything that says BADSTACK. When I say "fixing", I mean ensuring that every called function has correct stack displacement information. For direct calls, you fix these by editing the functions to make sure their purged bytes and frame pointer deltas are correct. For indirect calls, use ALT-K "Change SP value" to ensure the correct purge amount.