I see where you're going with this, but I'm not sure an XP "fee" is the way (or the only way) that I would go about this.

Perhaps stigmatize down-voting by publishing the number of downvotes (or a percentage of down/up) made by a user in their profile (color coded green, yellow, red?). Conscientious people don't generally want to be seen as assholes.

Also, the idea of requiring a reason for a downvote may in itself be enough to curb old downvote habits. It will let users from elsewhere know that this place is not like their old place.