And there is. What happens to that transgender woman when she's forced to use the men's facilities? Furthermore, I've never bothered to wonder with what genitalia the person in the next LOCKED stall is using to pee.
Transgender women are not interested in raping me. They aren't sexually attracted to girls and women. That's straight men – straight men who never seem to have learned to seek privacy to pee.
Moreover, even though we didn't have a name for them, transgender people have existed for about as long as humanity has existed. We've likely used the bathroom with them every generation since public facilities have been around. Again, we neither knew nor cared what bits the person in the locked stall next to us was using to pee.
Grow up and get your head and your politics out of other people's private bathroom stalls.