Executable relocations, whether performed for optimization or security, will only relocate the image (executable, shared object) as a whole.
For that reason, to bypass ASLR for example, any single address within a chosen shared object is sufficient. Given, of course, you know the precise version and build of the shared object. Knowing the specific build might be an issue by itself, however.
The reason relocations are done at the shared object level (and not, say, the function level) is because a shared object often has many internal relative references. Those are references that are addressed relatively (and not absolutely) within a single shared object.
In order to relocate at a lower level, many more relocation fixes will be required of the loader.
Moreover, and this is more of a historic reason than a technological one, relocations were intended to solve a problem with sharing an address space between multiple shared objects. There was simply no need to do more than change the location of a module altogether. The same base properties were later used for enabling ASLR.