You might want to read this blog post on this subject. What I'm quoting here is the central message, but do yourself a favor and actually read the rest and don't just respond based on this quote

Tolerance is not a moral absolute; it is a peace treaty. Tolerance is a social norm because it allows different people to live side-by-side without being at each other’s throats. It means that we accept that people may be different from us, in their customs, in their behavior, in their dress, in their sex lives, and that if this doesn’t directly affect our lives, it is none of our business. But the model of a peace treaty differs from the model of a moral precept in one simple way: the protection of a peace treaty only extends to those willing to abide by its terms. It is an agreement to live in peace, not an agreement to be peaceful no matter the conduct of others. A peace treaty is not a suicide pact.

When viewed through this lens, the problems above have clear answers. The antisocial member of the group, who harms other people in the group on a regular basis, need not be accepted; the purpose of your group’s acceptance is to let people feel that they have a home, and someone who actively tries to thwart this is incompatible with the broader purpose of that acceptance. Prejudice against Nazis is not the same as prejudice against Blacks, because one is based on people’s stated opposition to their neighbors’ lives and safety, the other on a characteristic that has nothing to do with whether they’ll live in peace with you or not. Freedom of religion means that people have the right to have their own beliefs, but you have that same right; you are under no duty to tolerate an attempt to impose someone else’s religious laws on you.

[…]

If we interpreted tolerance as a moral absolute, or if our rules of conduct were entirely blind to the situation and to previous actions, then we would regard any measures taken against an aggressor as just as bad as the original aggression. But through the lens of a peace treaty, these measures have a different moral standing: they are tools which can restore the peace.