Nobody wants her out because she is a woman or a minority. They think she is incompetent, and in their attempt to criticize her, they fall back on petty insults (which is definitely A problem, just not the same problem.)
I think of it like getting cut off while driving. You might be angry at the driver, and yell insults based on age, race, gender, looks, baldness, big nose, whatever… You don't hate this person because they are say, bald, but it's the easy thing to pull out and insult with.
People hate her based on – paraphrasing Pao herself – 'behavior'.