Windasm32 is a very old 32-bit program and is probably running out of the 2GB virtual memory space allocated for it by Windows (you should be able to confirm it using a tool such as VMMap). One possible workaround could be to set the
IMAGE_FILE_LARGE_ADDRESS_AWARE flag in the PE Header which would give it an additional 1GB of address space, e.g using
editbin /LARGEADDRESSAWARE windasm32.exe
However this may lead to bugs and/or crashes if the program is not prepared to handle addresses above
0x80000000. And it may still not be enough for the program you're trying to disassemble.
I would strongly suggest using another, more modern disassembler instead that can handle big files. If you can't find the functionality you need, ask a new question about it.