You're Canadian. Make that very clear, and you won't have a problem with anyone. So long as you're a decent person ofcourse.
It's just what we see of people. The Americans we're exposed to over here, are stupid, sickeningly patriotic rednecks. (What we see of Canadians is just nice, happy, friendly folk. Hence the "make it very clear you're canadian".)
But yeah, a lot of the accents are very irritating. They're just so whiny and nasally usually. And when they're not, they really don't help the whole stupid image with the way they sound.