I think for trans men it's a non-issue. If you aren't on testosterone you should be able to choose, there's no advantage. If you are on testosterone you compete as a man and should always be given a therapeutic use exemption so long as your levels are kept within normal male range. Trans men will usually be less competitive until they've been on testosterone a few years (like how teenage boys can't compete with men) but not much you can do about that.
I think with trans women, the issues are over stated. The hormone requirement is what science tells us is long enough for them not to have a 'natural' strength advantage. The Olympic committee didn't take this decision lightly, they wouldn't do anything that opens them up to accusations of unfairness. They knew they had to be able to justify it well if a trans woman does happen to win a medal. The fact they went with allow trans women in shows the evidence is there.
You do get things like trans women being on average taller than cis women so may be better at something like basketball. But why does it matter how she came to be taller than average? Why say it's unfair that a trans woman is tall if you wouldn't a cis woman who happens to be the same height? If there's a point it's that some people being taller is unfair on shorter people who want to play basketball so we need height categories. That's obviously a bit silly, especially at Olympic level.