Depends what you want to do. If performance is your thing, and you really want now to have a career in orchestral/operatic/solo performance or if you're really serious about composition and you do a composition course, then go for it. However you don't necessarily have to do undergraduate courses at conservatoires. If after doing an academic course at a normal uni while specialising in performance, it's possible to get on a postgrad course at one. A lot of singers go to a normal uni first to give their voice time to mature.
Bear in mind though the courses would vary a lot between normal unis and the music colleges. At music college, as one would expect, a lot of your time would be spent having tuition, practicing and playing in ensembles. You would have to be pretty devoted to it. At a normal uni, you could obviously do those things, but they would be balanced more with musicology, composition and other things. At a music college, there would be perhaps a more competitive atmosphere with so many talented musicians, many whom you would likely be competing with in the highly competitive music industry. Some thrive in it, others perhaps can't cope with the demands.
So in summary, going to a normal uni will provide you with a broader education and provide you with more of the skills that employers want yet don't expect to get as much performance tuition paid for. However, if you are determined to make a career out of performing/composing, then a music college may be better.
As for how difficult they are to get into, a lot of them prize themselves on having extremely talented musicians, and pretty much the whole application process is based on the auditions/tests. As a result, it can be pretty difficult to get in as they generally expect people to be grade 8 distinction/diploma standard or better. However, it can be worse for certain instruments than others.....voice (particularly soprano) and flute would be some of the hardest to get into I would imagine alongsides piano and flute just down to how many people play them to a high standard. If you applied for tenor horn or euphonium or something rarer like that, then you probably would stand more chance. Since they aren't predominantly academic institutions, it is common for many of them to give EE offers so you could say that they aren't that hard to get into really.....