Lately I've been experimenting with software-defined radio (SDR) and have come up with some heuristics by which I can visually identify certain kinds of signals by looking at their representation on a waterfall diagram. Certain obvious characteristics such as frequency and bandwidth can be measured, but how might one systematically go about determining such things as modulation and channel coding?
All of the books and papers I've read describe the process of transmitting or receiving a known signal, but none that talk about the identification and decoding of unknown signals. Is there a system or method? Are there papers, books or other documents that describe this?
Edit: Perhaps it wasn't clear enough from the original question, but what I'm looking for are techniques (read automatable techniques) that might be used to classify and demodulate an unknown signal that is received over the air. For example, by what method might one automate a method that would reliably identify 16QAM modulation?