Of course it's fair. We're not all born the same and it's our individuality which is what sets us apart. Extra time in some cases is essential due to the underlying medical condition of the student. Of course some students unfairly claim it, but if we were to ban it, a dangerous precedent whereby those with legitimate need would be denied.
If there is some great panacea to cure us all from every condition to make us all the same, throw it away. Our individuality defines us and it's one of the reasons I went into medicine: to see people of all types, ethnicities, ages etc. Many of my peers have extra time and frankly, if I've done my revision and worked hard I don't care if they get more time or not because at the end of the day you get a grade, try to make it the one you want because of YOUR effort YOU put in.