I think part of this comes from living in a society that is so steeped in a belief in a cosmic balancing system that rewards good behaviour and punishes bad behaviour. It encourages the belief that being "good" or "not bad" confers some sort of karmic points bank you can draw from later.
The fact of the matter is this: having a really great reason, a legitimate excuse, whatever, does not exempt you from reality. Nobody is going to give you an exemption from heart disease and diabetes because of your inability to exercise or eat wisely.
Having legitimate psychological reasons for repellent social behaviours is going to drive people away.
Your rationalizations for why this or that social more, legal restriction, or law of physics does not apply to you do not change the fabric of reality.
Even the belief in some sort of afterlife "balancing" of the books does not suggest that any sort of balance is to be expected on this side of the Veil.
This is not meant as criticism and is not directed at individuals. It is something that has been growing in me as I have lived and observed how we all rationalize our personal exemptions. It is a reminder to myself as well, since I am as guilty as anyone else of using my very good reasons to try and get myself off the hook.