“It’s okay because there are very few of them” was how I remember it being described as a child. Same with “It’s okay because they’re white” for why it’s okay to make Polish jokes.