Actually his children ARE young and dealing with their parents being apart, different homes, lack of stability etc.
Sounds like you have several problems, including communication. At the moment there's nothing stopping you contacting your boyfriend to find out the plan for today. For the longer term, you need to talk to him and decide what the arrangements will be. Why don't you see him during the week? Will you live together? Can you meet and get involved with the children at some point?
Jumping straight to breaking up with him points to a lack of maturity and forethought when getting involved with a father.