It is possible for an attacker to replace the file on the disk by beating the race condition. When you write the file to disk, there is a delay until when the function you called which loads the DLL actually reads it from disk. An attacker can reverse your application and find out where you place the DLL, then in a no-sleep thread (maybe even multi-threaded) application can constantly spam to see if there is a new DLL placed, if there is, quickly replace it.
Another way to prevent you from reading it instead of just writing over it as fast as possible would be to get an operation lock on the file preventing you from reading it until the oplock handle is closed.
Second, if an attacker is at the same privilege level as your process, he can easily tamper with your process. You do not need to attach as a debugger or "debug" to inject code into your process. All an attacker needs to do is OpenProcess your process, WriteProcessMemory his shellcode, and then CreateRemoteThread the shellcode.
This is only one way of injecting code, there are SEVERAL more and you can't ever be 100% protected. Returning to your question whether or not an attacker can hijack your DLL, they most certainly can. They can overwrite portions of your DLL to hook or do another operation.