No
I did it. I had an ex boyfriend who I thought I loved - we got on well enough but in the end we were just too different. This was why we broke up - the endless arguing and the idea of what my future would be if I stayed with him. He was always skint, so we could never do anything together, yet smoked endlessly. His job was next to nothing - he was offered a better job as a partner in a smaller job (it was car valeting, so very low requirements, but a friend was wanting to start up properly) and he turned that down. He never finished college because he just didn't turn up or do the work. It bugged me that he didn't want to better himself.
After that I went on to find my current bf. He has a good job that he loves and is determined to get his degree - doing a college course and now his degree part time. If I look at both bfs, they both had similar GCSEs and similar prospects, just the current one turned it around and wouldn't give in easily! We have similar goals and ambitions, so we agree on more things and don't argue about big life challenges etc.
At the end of the day, the decision is yours. I broke up with my ex because of the arguing since we couldn't agree on the big things in life, and that was a big deal breaker for me. You just need to decide where your priorities lie.