Yes, of course you can go into other colleges! You'll probably make friends at other colleges so you can go and see them. The libraries tend to be restricted (I know Trevs is anyway) as are the bedrooms (though obviously you can go and visit your friends' rooms) but pretty much everywhere else is open. However it may depend on the college...
You can go and hang out at other colleges, though if you're on your own at a college that isn't your own you may get some odd looks I guess. Given that's the castle I think it may be more restricted though, but if you show your campus card it'll be fine I guess? Other colleges are more open though.