That’s true.

I would’ve probably responded in a different way, though. You can simply say that the allegations are untrue/silly and you won’t waste time arguing them. Otherwise you’re lowering yourself to the level of the attacker.