We all slip up from time to time. If it happens only once in a while catch a taxi and let it slide, if it starts to form a trend explain to him that this is a deal breaker for you and that you're unsafe, you don't just feel unsafe but you physically are unsafe. From the sounds of it you're still at work so it's not like you're waiting at the bus stop only to find him plastered suddenly so you have a chance to find an alternative method of safe travel.
Honestly, he probably doesn't realise exactly how much more dangerous it feels for women to walk home at night. In my experience most guys tend to have a false sense of security about this, I know I did. I used to walk home every night through a large, poorly lit park with the highest crime rate of any public area in the city. Mostly I would walk home between 11pm and 3am on my own. A couple of guys and a knife made me realise my mistake.