Workload will be pretty much identical between colleges but will depend on your choice of courses: Your first year is the only one without choice so you can use it as a benchmark. For this, you do 8 courses: 4 per term for 2 terms with Easter term reserved for revision and exams. Each first-year course has 24 lectures and 4 example sheets spread over 8 weeks. This works out to about 2 example sheets per week and 12 lectures (which are about 50 minutes to an hour) per week. Example sheets can take some time, it's hard to say exactly as a lot of it is going over stuff in lectures and many of the questions are fairly approachable once you understand the content, which can take more time than actually doing the questions, but between 4-12 hours per sheet would certainly describe most of them. Often, the last example sheet bleeds over to the next term as you need some time between completing the necessary lectures and doing the problems.
In later years, you can choose fewer or more courses but expect to have a fairly similar workload. In second and third year, there are also 16 lecture courses with 2 lectures per week and 3 example sheets per course; in the second year there are 12 lecture courses with 2 lectures per week (they don't run for the last 2 weeks of term) and 2 example sheets per course. You can see on the schedules exactly how many lectures each course has. I think the majority still have 24 lectures and work as in first year.
In the second and third year, there is CATAM, which is the computational coursework. It is technically voluntary but is set up such that almost everyone does it in the holidays.
Regardless of college, you will attend the same lectures, have the same questions on the same example sheets and do the same coursework. Your supervisions will be organised by your college but, especially after first year, will often be done by supervisors from other colleges and be pretty much the same as those who attend other colleges. That is to say that your workload won't meaningfully differ between colleges.
I am, however, at Sidney Sussex and I'm happy to answer any other questions about it. I have some experience with Catz also, though not too much with Peterhouse.