He might be able to do that, but I don't see how that would work. Why not just add you to another group?
A lot of people seem to not like working in groups - it's an important skill to have so to be honest they should really embrace the challenges that come along with it.
On the other hand, if you are working on your own, and it's a small project, you should be okay. If it's not practical or reasonable for you to be on your own, then you should inform higher ups.