Do universities only care about you 3 best a levels grades?
In general, yes. Almost all offers are based on 3 A-levels.
Two points...
Some offers might well specify which A-levels they want the grades in. If you were applying for Physics with A-levels in Physics, Maths, Chemistry and Art then the latter will probably be excluded.
The only 4 A-level offers I have heard of have been at Cambridge and only rarely.