The difference being one is not derogatory. It's the difference between saying 'all black people are intelligent' to 'all black people are stupid'. Both terms are based on relative intelligence of all races, but only one is specifically offensive to a certain group.