Hybrid vigor also means you can mix with someone a few villages over and it fixes any potential genetic "problems" you have.
You don't have to mix races, you just have to make sure their generic heritage is ~30 miles from you to get the most benefit for your children. Our world wasn't always as interconnected as it is now. Not to mention it doesn't really fix racial disparities for diseases.
For example, whites are 5x more likely than blacks to get testicular cancer, but on the flip side, blacks are 5x more likely to get prostate cancer vs whites. You get a white/black hybrid and they don't tend to have reduced chances of either.