Technically it is both.
The primary reason is due to the change in refractive index, which causes a change in speed. However the refractive index of a substance is different for different wavelengths of light. This change in refractive index is small but it exists. The refractive index of a substance is higher for shorter wavelengths of light. Therefore, shorter wavelengths of light do refract more due to slowing down more.
For example, when we say that the refractive index of a substance is 1.5, that is really only an approximation of the refractive index. The actual refractive index might be something like 1.503 for red light and 1.507 for blue light.
So in conclusion, the actual reason for refraction is the change in speed of light, and it just happens to be the case that shorter wavelengths of light slow down more, because the refractive index of substances are actually higher for shorter wavelengths of light.