Is it bad to not want to see my brother as often?

I’m 23m, brother is 37

I don’t have much or anything in common with my brother other than we are related.

My brother lives in a flat and doesn’t work or do anything. He receives high rate of benefits so that is how he lives and that’s fine with me

Grew up with mentally physically abusive parents and got kicked out at 18. So I was living permanently on my sisters sofa until 2021 until I got into uni

My brother wouldn’t have me in, no reason other than he doesn’t like sharing his space with anyone. Even his own homeless brother at the time. I spent 1 night with him and he admitted to my sister he wouldn’t let me live with him. So if it wasn’t for my sister I probably wouldn’t be here

So that’s fine. Me and my sister are pretty close. My brother and I it doesn’t feel like he’s my brother, it feels like a mate relationship.

Times were I stayed with him he ate dinners in front of me while not offering anything, if he had food for himself he wouldn’t make you lunch or dinner because that’s his food specifically for a set amount of days

Last year I was out for a walk with him on a warm sunny day, he started to show off drinking (when I was a kid at around 11 he used to down vodka and beers in front of me, surprised I don’t drink) - he drank before he eft his flat, took a photo of the alcohol and sent me it on Snapchat with no caption. Then we went on a walk and then he went into the off license, bought a quarter bottle of something

Then since I decided I didn’t want to watch him drink I decided to meet friends instead, he then went on a rage and text my sister that I am boring and that there is ‘more entertainment in my left toe nail’ lol

I since have forgot that with him and don’t mention it anymore, but I still remember

He just doesn’t have concern for anyone, even his own sister. So I prefer to be away from him

He reminds of our toxic parents, which kind of angers me and I don’t feel so good

So, last minute plans, on Friday I was meant to stay in my sisters,
Her boyfriend decided to visit and stay until Tuesday, so I have to stay up with my brother

However, I’m planning to leave back to university tonight or first thing tomorrow morning as I struggle to stay with him.

If her boyfriend decided to visit, which is obviously ok with me and he’s a cool guy, and I have to stay up with my brother. How do I decline this without sounding arrogant?

I honesty don’t hate my brother, I’d just prefer to spend less time around him.