Simon Says, But With Servos

How much easier would life be if you could just grab hold of whatever mechanism you wanted to manipulate, move it like you want, and then have it imitate your movements exactly? What if you could give a servo MIDI-like commands that tell it to move to a certain location...