Id say you have a right to be upset by the stereotype everyone's openly displaying, but i feel as though you should just ignore it/brush it off.
As goes for the secondary school friend of yours, i doubt s/hes aware that you live in a council estate and you would feel offended. S/he probably just invited loads of random people from their friends list. So dont feel as if inviting you is an attack on youpersonally.
We all learn to live with stereotypes, but they are hurtful, so youre not being too sensitive, youre only being human.
You could always tell the person your opinion, but to be fair, i doubt he cares and will probably continue going, considering you barely know each other.
Id just say brush it off.