I just want to be with him.
That is extreme short-termism.
In the medium to long term, you really really DON'T want to be with him. You want to be with someone that can handle stress and conflict far better than him.
The day that you split up with him, which sounds like it was yesterday, will be the best day of your life so far!
Congratulations! You have an escape route from this moody, rude, abusive, inbalanced man.
Please make his wish come true! The one where he said he doesn't want to be with you again. Make that your parting gift to him.