I thought I heard a voice at night when I was really young. It kept saying "doom," in a threatening voice, and I didn't like it. I eventually found that I could control the voice... it would read anything I was reading, and then as long as I could keep my mind focused on forming thoughts as if I were going to speak or write something, I was fine and the voice was compliant. It would happen whenever everything was quiet. I would hear a strange, constant tone start in my ear, followed by the voice. Music, television, or conversation were enough to stop it from happening, and it tended to appear whenever I felt anxious and there was no sound. So I just started leaving the lights and television on all the time until I got older and it stopped happening.
If your situation is anything like mine, the reason you're hearing a voice telling you to leave when you're around people is because you feel unwelcome/scared on some level of the situation. You may actually be hearing your own inner dialogue as an external voice because you feel overwhelmed.
Still, if you can't gain any control over it yourself and it keeps getting worse, you might need to see a doctor.