don't pay him a penny. unless you signed something to say you would be liable for any mishaps, then you aren't liable.
on a moral basis, you paid him for use of the car, this includes insurance on the vehicle, if you crash, cause injury to someone, damage the car or others property, the insurance will cover it.
he just doesn't want to go via his insurers, the excess is probably higher than a standard vehicle, and he will probably lose his no claims, but that's not your problem, its a cost of his business.
There's no being a 'douche' involved, morally and legally speaking you don't have to pay a single penny, even if you were liable by something you have signed, the extent of your liability would be to a value of the excess that his insurers charge.
Just say you're not paying for it, if he wants to pursue it further he's welcome to take you to court.