You should check the EFF RE FAQ. It might help you wrap your head around some of the legal aspects and definitions.
From where I see it, yes, it is reverse engineering. But, it's not clean room reverse engineering.
There are many layers/levels of RE. You can target the hardware, the firmware, the software, the algorithms, the architecture, ... The higher you go, the more abstract the construct. So, depending on what abstraction you're trying to model from the available elements, the RE tools and methods will be, somewhat, different. But, it's still RE.
I would define RE by the attempt to build a higher level construct from a set of packaged, or obfuscated, or low level elements.
If you start from a binary without any prior knowledge of its inner-workings and dig your way into a source code, that's clean room binary RE for the intent of producing a source code.
If you peruse (in the true sense of the word) the source code and manage to identify the algorithms (some homemade implementation of FFT, AES, or some scheduling scheme), that becomes a clean room algorithm RE.
If you have prior knowledge of the source code and you manage to identify the algorithms, that's algorithm RE. Is this case, the manner in which the source code was obtained is crucial in defining the legality of the RE work and the intent of the RE.
If you are learning by playing around with prepared labs/exercises, well, you're working out your RE skills.