A rule of thumb I was told in my first year of university was "make sure you have at least 6 sources". Of course, once you find 6, you end up finding more anyway, but the principle is that you really shouldn't use just one source, otherwise it's just a literature review. It's annoying but also great that you've found that one perfect source but you do need to find more. What I tend to do when I want to reference a point but I feel I've used that book too much already is find another source that agrees/makes the same point.