I've read a few scientific articles, including on the NIH website, that detail how there are multiple "gender assignment events" in utero. It's never been a cut and dried only the genitalia determining gender assignment event.
A couple of those reports are here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6677266/ and here https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7031197/