Assuming the you are speaking about strong encryption, most of the algorithms are supposed to be indistinguishable even if you provided either the key or the clear text. So, it should not be possible to know that a given key is used just by looking at the result.
One example of this is the field of Kleptography where:
[...] the outputs of the infected cryptosystem are computationally indistinguishable from the outputs of the corresponding uninfected cryptosystem. Hence, in black-box implementations (e.g., smartcards) the attack is likely to go entirely unnoticed.
As you may have noticed, the Kleptography is safe only on black-box implementations, so there indeed room for detection in white-box attacks.
But, I have not enough experience in this topic to give you general advices about it. Except the fact that you should start to look at real World implementation of cryptographic backdoors before trying to detect it. You may want to explore some of these links:
And so on...