You've posted a lot in this thread and I've tried to aggregate my responses into a single post. I think, and correct me is I'm wrong, we disagree on the point of what powers the right to not listen grants us. You seem to be confusing listening and speaking.

I think your above analogy fails in the following way:

would you be as insistent that a group of friends keep a person in their circle of friends even though none of them like that he is really rude to everyone else, simply because it's his right to express his opinion?

Absolutely not, nobody is saying you should be forced to listen to things you don't like or hang out with people you hate. It is a personal decision that everyone can make for themselves: "Based on his comments, I think Shiranaihito is an asshole so I will not read them." and the other is imposing your ethics on someone else "I and my friends think Shiriranihito is an asshole and should have his ability to express his ideas removed." In your analogy it would be more like Guy X is an asshole so lets not talk to him, vs Guy X is an asshole so lets not allow him to be in the same building as us.

Down in the comments you say:

Sure the terms racist and misogynist etc can be misused but when people are not being hateful, but you are also essentially doing the same thing by assuming everyone who points out racism and misogyny to be wrong. My point is when those legitimate cases arise it is also our right to not want to listen to them. No one is saying they don't have the right to say that stuff, we are saying we don't want to listen to them say it.

It is absolutely your right not not have to listen to that stuff, but there is a difference between listening and speaking. I don't listen to the crazy religious people on the sidewalk when they shout their drivel, I just ignore them and walk by, maybe if they go too far I might say "dude, that's not cool." That is exercising ones right to not listen to bullshit. Going to the city and passing an ordinance that their group is now barred from shouting stuff on the sidewalk is not exercising ones right not to listen, it is forcing them to stop expressing their ideas and is wrong.

Another example: I think BDSM is gross. When I go to porn sites, I don't click the BDSM tab. Occasionally a BDSM video makes its way to the front page, I see that it's there but don't watch it. That doesn't mean I think BDSM should be banned on the sites I visit.

You also make a chess club analogy:

Are you then saying, that you are not ok with say, people creating a site designed for discussing chess and then asking people to leave if everything they post on that site has nothing to do with chess what so ever?

Nobody is saying that someone who only talks about football rather than chess should be removed from a chess site. People are saying when a site is designed for the sharing of ideas and creation of special interest tribes, that to allow someone to make a tribe for chess but not allow someone to make a tribe for football is wrong.

The Following are my ideals on free speech and expression: If the KKK were holding a parade in my town, I would not go to it because I think they are terrible people. I would not however say they are barred from having a parade in my town. Freedom is a very delicate thing, if I say "I live in a free society, but do not allow the KKK to hold a lawful parade because they are terrible people." then I am lying and I do not live in a free society. Idea...