When I learnt to drive (8 years ago, mind you), on my first lesson I did maybe 75% of the driving, and he did the other 25%, and for all the other lessons after that I did all the driving. I can understand this isn't how every instructor would do it, but 4 lessons in I would expect to be doing lots of driving.
If you've spoken to him and he's just shrugged you off, then I suggest voting with your feet and changing instructors.