Generally, it's a complex topic. There seems to be very little in the way of example or linear progression in to non-trivial examples.
It's possible my google-fu is weak, but I can't seem to locate decent tutorials on using binary instrumentation frameworks (Pin, DynamoRIO, other).
What resources could someone who is interested use beyond stumbling around until they get it working?
After some of the answers, I thought I should tack on that dynamorio.org is no longer responsivesometimes non-responsive. The project is on googlecode here.