do you have another lecturer you can trust/talk to who you can share your concerns with? if you talk to someone you already get on with, perhaps they can ease your mind and direct you in how to arrange for a different personal tutor. it doesn't seem like such an unreasonable request if it's that uncomfortable.
i would bet tho the last thing your personal tutor wants is to have an uncomfortable meeting and will do whatever it takes to ensure it doesn't happen. just whatever you do do, do not avoid their emails or the meeting. i was in a very similar situation to you myself and can say avoiding the meeting did not turn out so well as they start to worry.