Do go back to college. It's easy to blame everything on the past but everyone has a 'past'
If 50 somethings can train from scratch to run a marathon you can learn and be a late starter
Ask at your local college as many offer free tuition in Maths and English. I believe they should offer you an assessment, so just go down there and see how you get on?
You won't be alone and if you don't do this now you never will ..... what is to lose?