Assuming this is all true I would say in such a scenario he is most likely attracted to you. However you must understand that for a man in such an office environment it is really difficult to 'make a move' as it were. Basically a lot of guys in the workplace particularly the office environment are sh*t scared of complaints of 'harassment' well any decent ones are. The ones that aren't just don't plain care and will just steamroll right in. Anyway, hence the beating around the bush that ensues while the guy tries to see if there is someway to negotiate he's way to a date with accusations and being fired from his job.
So it depends if you are interested in him? If you are not bare in mind the poor chap has some form of a life that might still be worth living and likely needs that job to pay the bills. So the nice thing to do is to let him know subtly that you aren't up for relationships at the moment and if necessary distance him in the workplace if it makes it easier for you both. If you are interested in him then suggest a casual meet up during lunch or after work. It will be way easier for you to do this than him as basically guys rarely ever make formal complaints about females coming onto them and even if they did they tend not to be taken that seriously unless the female is basically relentlessly hounding the poor guy without respite when he clearly doesn't wish for her.