Genetics in Limited Pools

So I was just considering the notion of the 4th Y again, and it is making me consider the nature of genetics, specifically when the pool of available partners in a species is limited or singular. Whether one considers this from a biological/evolution perspective, or from the perspective of Noah and his family stepping off the ark, at some point in human history there would have been a time when one's family would be the only available partners available for sexual reproduction. Given how such pairings work now, where offspring quite often become far less suitable to their environment, often with what we would call "birth defects," this must mean that at these junctures genetics would necessarily work differently than how we see it work today. If random chance is the governing aspect to how genes are combined and expressed today (certainly reasonable from a scientific understanding and confirmed by experimentation), then this would mean that at these junctures it could not, by any stretch of reasonableness, be random chance that governed the process. While it is possible for a moonshot baby to come from those directly related, where they have no defects and perhaps even end up better suited for their environment than their parents, consider how many times this would have to happen before you would end up with the variety of available and genetically dissimilar partners seen in a stable species, where mating with a partner who is distinct from yourself and who is suited for their environment would be the best chance one's offspring have for success in the same environment.

For this reason there must be more going on with genetics, and I am imagining a kind of codex (as I have imagined before) written on the level of DNA, a kind of guidebook beyond what I understand is generally understood, or is at least accepted, today. This may be on all Y chromosomes, this may be hidden somewhere on Xs/all chromosomes, but the fact remains that this guidebook must exist, for sexual reproduction to occur successfully enough within limited pools (a pool which would have necessarily occurred at one time for every currently living, and historically recorded, successfully sexually reproducing species), to avoid the genetic pitfalls that we know occur in such pairings today. I don't know why I'm just really connecting these dots today, and I'm curious why this is not a topic I learned about in biology in school. Is it acknowledged? Is there a proposed solution from a purely scientific perspective?