You can't truly optimize yourself as a person without optimizing your body as well as your mind. I dislike 'gym culture', but I also envy the 'douchebags' who have the self-discipline to get up every day and go through the physical pain of lifting in order to improve themselves.
For most people, there is no excuse for not being both at least a little intellectually and physically capable.