The maintenance of polygenic sex determination depends on the dominance of fitness effects which are predictive of the role of sexual antagonism

In species with polygenic sex determination, multiple male- and/or female-determining loci on different proto-sex chromosomes segregate as polymorphisms within populations. The extent to which these polymorphisms are stable equilibria is not yet resolved. Previous work demonstrated that polygenic sex determination is most likely to be maintained as a stable polymorphism when the proto-sex chromosomes have opposite (sexually antagonistic) fitness effects in males and females. However, these models usually consider polygenic sex determination systems with only two proto-sex chromosomes, or they do not broadly consider the dominance of the variants under selection. To address these shortcomings, I used forward population genetic simulations to identify selection pressures that can maintain polygenic sex determination under different dominance scenarios in a system with more than two proto-sex chromosomes (modeled after the house fly). I found that overdominant fitness effects of male-determining proto-Y chromosomes in males are more likely to maintain polygenic sex determination than dominant, recessive, or additive fitness effects. I also found that additive fitness effects that maintain polygenic sex determination have the strongest signatures of sexually antagonistic selection, but there is also some evidence for sexually antagonism when fitness effects of proto-Y chromosomes are dominant or recessive. More generally, these results suggest that the expected effect of sexually antagonistic selection on the maintenance of genetic variation in natural populations will depend on whether the alleles are sex-linked and the dominance of their fitness effects.