I lived in Toronto for a year (lived before and now live in a small town in the South of England) and so I'd obviously recommend it! It has a ridiculous amount of things to do and something for everyone! I'd recommend the zoo (pandas), Casa Loma if you're into History, the CN Tower, catch a ball game if you can, the Toronto Islands are beautiful and one of my fav places, the Eton Centre is pretty good for shopping (although it's a lot less good then it was 7 years ago!), the waterfront is lovely for a walk in the summer and there's an AMAZING ice rink in the winter and I'd also recommend just walking around the city. Like much of Canada, it's incredibly culturally diverse and there's a great many 'little' neighbourhoods ('Little Italy', 'Little China', 'Little Poland' etc). There's a restaurant called The Madarin which is an all you can eat Chinese buffet which is incredible. It's has legendary status in our family, even all these years later! If you do get a chance, do visit 'Little Poland' (aka Roncesvalles) and say 'hi' to the Pope from me as that's the neighbourhood I lived in!
This summer we visited Quebec City which was an experience in itself. You know that you're in North America but it feels so so much like a European city! It was a very strange place (don't get me started on the 'French culture' and Bill 101) but had lots of touristy things to do.
We also visited Ottawa. I loved the parliament buildings and there was a very good war museum but it's definitely a working capital and I doubt there's enough to do for 4 days. We spent an evening and a full day there and I think we covered everything. If you go in the summer, they project a light show about Canada's history onto the Parliament buildings and that was completely stunning (and made me very proud to be Canadian).
Montreal I visited back in 2010 for a long weekend but we also drove through it (while driving from Quebec City to Ottawa) for lunch. After Toronto, I'd say it's your best bet (out of the places I've visited). There's lots to do (they've turned the old Olympic park into a sort of mini Eden project) and there's enough French to make it a bit exciting. As long as you're not a veggie, I'd recommend Schwartz's for a smoked meat sandwich. It's only a tiny place so you'll probably have to queue to sit down (but it's Canada so everyone is very friendly and you'll have plenty of people to chat to heehee) but OH MY was it amazing! It's pretty famous and has been open for years and years and I can understand why!
I haven't travelled in the west so I'm afraid I can't help you there. But Prince Edward Island gets an honorary mention because it's BEAUTIFUL (although not much of a city) and is amazing for anyone who's a fan of LM Montgomery! Sorry for the crazy long post, I seem to have gone a bit crazy whoopsie! Have fun wherever you go!