@GreyWolfGreyWolf is making a valid point that, looking back through history, we can find LOTS of things that have been used to protect elevated societal positions. In the area which we all love - using the English language - the Normans used French as the official language when, as a tiny minority, they took control of England. It served to cut those who spoke Anglo-Saxon and Latin out of the power structures.
Of course, there are those who would claim that teaching grammatical English today is racist, in much the same fashion as in Norman England. Don't teach English is one response. Teach those who need it how to break the code is another. It is, and has been for decades, a big fight in K-12 Education. See, e.g., Jaime Escalante, Marva Collins, E.D. Hirsch, Jr., and others on the "teach, teach, teach!" side.
Yes, GW, some of this is silliness, but the point isn't abstract arguments, or even sociological or linguistic studies - it's about power. (Most things in politics and law are - power is the universal human value that's most difficult to reconcile with family, friendship, and care for others. And, no, that's not my opinion. It's science, at least as far as the study of humans can be science - see Schwartz, Univeral Human Values.)
The point of the "math is racist" crowd is NOT to make a historical argument. It's to gain power today. Spots on corporate boards. Distribution of public funds to selected organizations. Those types of things. It's not about yesterday, it's about today and tomorrow. The argument of the "woke" crowd is that guns should be used to make the world a better place by forcing the results in the world that they want.
The real fight isn't over math. It's over whether our current society is one where individuals who do are not white males can exercise autonomy and personal agency to reach their personal potentials, build functional families (for those who agree that this is important), and live happy, satisfying lives. And, if not, who should we point guns at (a/k/a, exercise the power of government over) to achieve a better world.
Arguments over whether 2X2=4 is racist are usually a smokescreen or distraction from the real conflict.