Allow me to, well, play my own advocate here.

Even with your Master when you first met? He put the circlet on you. Used it to hurt you. You were charged with protecting someone you didn't know. And you protect those you care about. Yet with him, you had to protect before you cared. Was there not, even a small part of you, that was scared to open up to him when the journey started?