You were the 'top' and so the HIV risk is less.
If he's having casual unprotected sex, there are other things he could have.
If this is with 72 hours of the sex, get down to A and E now and ask about 'PEP', a month long supply of anti-HIV drugs that will reduce the chance still further.