The sex is determined by the chromosomes of the male sperm cell,
at the time of conception. However, development of a
fertilized egg is the same for male and female, until
at a certain stage, due to the genes on the
Y-chromosome, male fetuses start to produce substances
that change the development to follow the 'male'
course.
In fact, you are hitting a difference in 'genotype'
versus 'phenotype' here. The fertilized egg has the
genotype of male or female, but the phenotype (the
'form' you see) is not yet apparent. The difference
between genotype and phenotype is crucial in biology
but it is often ignored. You have just identified a
beautiful example.
-------------
sex is determined genetically at conception
by the inheritance of either two X chromosomes (female) or an X and a Y
(male). The expression of the sex characteristics doesn't begin until about
6 weeks gestation. Before that time the fetus is not really male or female,
but non-specific. The fetus has two non-specific gonads and two sets of
tubes. If the fetus has a Y chromosome, at 6 weeks a gene called the SRY
(sex determining region of the Y) turns on. This gene causes the
degeneration of the female characters and causes the male characters to
develop. It causes the non-specific gonads to become testes and the tubes to
become the vas deferens. Once the testes are mature enough another gene
turns on to start producing testosterone. This causes the internal sex
organs to develop. Part of the testosterone turns into another hormone which
then causes the external sex organs to develop. If the Y chromosome isn't
present the female pattern of development occurs; the female pattern is the
default pattern so-to-speak. The gonads become ovaries and the tubes become
fallopian tubes. So even though the gene for maleness is inherited at
conception, the expression of the trait doesn't begin until about 6 weeks after conception