The earth isn't of uniform density. That's for certain. You have liquids toward the centre with densities far beyond that of a rock on the surface.
The speed of the wave depends heavily on density. In general when it comes to crystalline materials and whatnot like a lot of rocks and metals, the harder it is, the faster waves travel in it, to my awareness.
Refraction implies a density boundary for the wave, i.e. high to low, low to high. That changes the speed based on the density argument above. Like glass and air are optically dense and rare, with light being slower in glass than air, a rock is "acoustically" rarer for sound than dirt, which is denser. Using density here in terms of the wave properties. Anyway, rare means that the wave travels faster in it. That's the basis for refraction so I assume using it in that sense is fine.