I don't see a problem with not upvoting something that doesn't interest you. For me, an upvote is usually good/ interesting + something I care about. It's not a bad thing to downvote, you just want to make sure you do it for the right reason.