That's the so-called 'paradox of tolerance' - https://en.wikipedia.org/wiki/Paradox_of_tolerance

"If we extend unlimited tolerance even to those who are intolerant, if we are not prepared to defend a tolerant society against the onslaught of the intolerant, then the tolerant will be destroyed, and tolerance with them."