An 'X' category defeats the purpose of the M/F identifier on the document. If information like that can be voluntarily withheld, why even have it on a passport? Regardless, sex is quite relevant when identifying somebody, as is a photograph. That's why it's on there.
People without a clear biological sex are incredibly rare and aren't good justification for essentially nullifying a relevant identity component on a passport (if it's open to everyone). In these cases, these people should simply pick M or F. It's not like it's a tattoo on their forehead; it's a document few people are ever going to see.
Some countries have already done this (if I remember correctly), but did so in response to political lobbying rather than sensible, constructive criticism of the passport identity system. It hasn't improved passports. It basically compromises important information and makes identity fraud somewhat easier, all to make a tiny minority of people happier with their passports.