This is how I typically do it, although it largely depends on the target/project and it's only how I like to do it myself:
- Start (probably dumb) fuzzing the application as soon as you have the target.
- Meanwhile, statically analyse the application in order to understand how it works and, maybe, to find low hanging fruit vulnerabilities.
- Try to understand if you can make any smarter your fuzzer once you have enough structures and functions discovered in IDA.
- If you have a hit in your fuzzer, discover the root cause of the problem and if it could be exploitable. If it isn't, nevertheless, it's worth checking in deep the area where the crash happened as that crash is a probable indicator of an interesting area to focus on.
- If you have a hit while doing static analysis, try to write a simple trigger for it.
- Focus on the part where the bug exists to continue statically analysing.
- Once you have 1 true vulnerability found via either by fuzzing or statically analysis, write rules/scripts to find similar vulnerabilities.
- Possibly write an exploit for it and go back to 2.
- When tired of finding such vulnerabilities and after understanding the inner workings of the application, try to find logical flaws.
PS: Reading documentation is also a good way to find some vulnerabilities, as well as reading change logs, diff patches, commit messages, etc... if you have access to source code (sometimes you can have partial access to the target's source code, even for closed source applications).
Just my 2 cents.