I've known people to. There are a lot of people who go into social work, having worked as Social Work Assistants, or other roles relevant to social work, for many years.
The universities will look at your previous experience, but it would be useful if you can also demonstrate that you would be able to meet the academic requirements (e.g. a good standard of written work etc). They usually ask for at least a grade C at GCSE in English and Maths (or equivalent) and often require you to complete a short written test prior to making an offer of a place.